I recently had a conversation with friends regarding some of the ways we use AI in training.  

Their questions were very pointed. My friends wanted to know what the ROI would be if they decided to use AI. I tried in the simplest terms to come up with some examples of how to describe the ROI and help others understand how we use it. 

I think the best analogy would be forecasting. In terms of safety training, the data has always shown that the longer a worker was on the job the higher the risk of accidents. When training was added, the data showed a drop in the frequency of accidents. All of those things have been proven over the years.  

Using AI, we will be able to collect efficacy data, i.e., did a specific training decrease the probability by a large percentage or by just a small percentage. How effective was the training in abating accidents? How severe were the accidents if any occurred? Would additional training lower or lessen the risks? Not only will AI track these things, but it will also be used for analyzing xAPI data from the client LMS to determine if the training was effective in achieving the client’s goals.

In doing so, we will be able to forecast safety issues before they became a problem for the client. In terms of the human cost, we will be saving individuals who could experience life altering or life ending accidents. This is just one of the examples of how we use AI in training. Using surveys, xAPI data, and observation, we can determine if the money spent on training helped the organizations reach their goals.  

It completes all four levels of Kirkpatrick’s evaluation model. Kirkpatrick might be a little too deep for a simple blog post. Hopefully, I have time to dive off into the topic at a later time. 

Let’s be honest, everyone wants to be a rock star or wants that recognition for succeeding. But, how do training and learning managers/leadership demonstrate real return on investment? Our teams are busy designing, developing and implementing training/learning programs. It takes time to provide real quantifiable data to prove results.

The Data

Think about the past two decades, how has the system for gathering and analyzing training or learning data changed? Or, should we ask, has it really changed? Is your organization still relying on surveys and learning management system data for the bulk of the knowledge gained for influencing and prioritizing how instruction or instructors should adjust or make changes to meet learners’ needs and expectations? 

Surveys are wonderful tools. We receive them in email, get asked to answer a few questions when handed our receipts and on and on. These simple tools provide details about wants, needs and performance. However, analysis of these simple tools has proven to be time-consuming and biased. Consider for a moment this hypothetical situation. A training director, who is preparing for an annual presentation, is reviewing surveys and notes a repeated complaint that there is no online training for a particular topic. However, the director is aware that the new online topic will be released next week. The director decides to omit this complaint because it brings the overall performance rating of the training department down.

Despite the learner’s age, occupation (if applicable), location or learning mode, the technique used did not have an impact on the results. Similarly, as technology evolved to allow less bias these factors seem to have a less impact on the data gathering and analyzing techniques today. Data collection began and continues today at times today with paper-based data collection. This can be time consuming and lead to missed data or misinterpretation. We admit that AI can be biased if the programming is not observed or verified for inconsistencies. However, our team of analysts review the AI program repeatedly and consistently to ensure unbiased results.

AI – The Evolving Solution

A solution that is being used outside of the training and learning industry is Artificial Intelligence. Although you may be using it in some ways, such as tagging data, it is most likely not being used to its full potential. To reduce training team resource strain while providing quantifiable ROI data, AI can be used to gather data from multiple sources. These sources include surveys, LMS, open source (such as employee sites or student sites), financial reports, HR reports, etc.). With the data imported from the multiple sources, our team of analysts determines what themes are appropriate based on parameters and conversations with our clients based on their needs.

As a result, AI combines with learning analytics and existing facts leading to more data to be potentially collected and analyzed at a more expedient, efficient and effortless rate than previously for your training team. Perhaps more importantly, your team will know the metrics presented for the training ROI is data driven.

You Succeed How?

While these details sound interesting, how does it help you succeed? Well, again your team is free to do what they do best – design, develop and implement training programs. You are given a customized interactive dashboard that allows you to drill down from the organization to the employee/learner level. This not only allows you to provide ROI for your organization senior leadership and prove the training value. It allows you to set achievable priorities and goals based on the identified needs in the data. More importantly to the organizations senior leadership, you will be able to align these with the strategic action plans. 

Interacting with the Dashboard

While this may sound interesting, we are sure it would probably be more interesting if you could actually interact with an example dashboard and see the results. Well, now you can. 

We have an example dashboard, Learner Analytics – Professor Comparison, available for exploration on the open Tableau site. Although not all our capabilities and customization possibilities are displayed, it will provide you with an opportunity to explore.

This dashboard was created based on a fictional university with two professors. We considered open source (ratemyprofessors.com and university survey data) along with grades, demographics, etc. in designing this. As you explore, consider how this could relate to your learners and/or employees. 

Remember, this is only an example of a potential interactive dashboard. Our dashboard can be customized to suit your needs whether your organization is higher education, medical facility, government or industry.

More Questions or Need More Information?

After reviewing the dashboard, feel free to contact us if you have any questions, would like more information or would like to schedule a demo for your organization. 

When training employees, you work with details, procedures and facts. This provides your learners with the knowledge and capability to successfully accomplish their jobs. As you start to use AI in training and analysis, do you have “all the facts”?

In this Ted Talk, Matt Beane speaks about how surgical residents train with and without AI tools.  What lessons learned apply to all industries from his insights?

AI + Learner

Consider Mr. Beane’s Ted Talk, not everyone uses AI to teach. We train using eLearning, VR and augmented reality solutions. These methods involve similar issues discussed in the video: an inability to “see one, do one, teach one”. Training teams develop the materials, present and gather data from observation and/or assessment(s). Most of the time, your training teams gather these data sets from the learning management system (LMS) or surveys.

Moving away from the expected LMS parameters, training teams use AI within an LMS to tailor for individuals. Each learner receives course suggestions based on past performance, enrollment or job. This is similar to Google Ads, Netflix, etc.

Let’s go another step further, are there questions and answers hidden in the data? If the data or facts are limited, AI learning analytics adds a new dimension to your details. Consider what the eLearning Industry discussed as the importance of AI to L&D (training). Moving beyond the algorithms within the LMS, AI offers an ability to “explore limitless possibilities”. Of those surveyed, almost 85% believe their company will gain a competitive edge from AI. Why?

AI + Learning Analytics + Facts = Training Future

From a training survey perspective, one option often ignored are open-ended questions. Although these provide more insight than a simple likert scale, the answers are time-consuming and inefficient for training teams to decipher. However, an AI could be designed to target words and themes which would allow the training team to gain the knowledge efficiently, expediently and effortlessly. Perhaps more importantly, your team will know the metrics presented for the training return on investment (ROI) is data driven.

To learn more about an AI training solution for your organization, read our whitepaper or view the AAILAT demo. We offer a customized solution. Contact us to schedule a time for a demo and answer questions.

When was the last time you completed a puzzle? Was it a landscape scene? Did it have similar colors? This may have made it more difficult to distinguish between one piece location and another. Did you lose objectivity of the whole picture from a single piece?

The same is true as we begin to look and review data for ourselves. We may have difficulty discerning between the important pieces of information or determining how one piece of data relates to another. This can become even more complicated with additional data.

AI’s Objectivity

Due to its objective and program abilities, AI is being used in multiple industries with success. Recently, Artificial Intelligence, published an article listing the top 10 businesses using AI. These include marketing, sales, human resources, and customer experience, to cite a few examples.    

Each of these industries are using AI for data analytics and big data to gain insights and improve an overall set goal or objective. In this capacity, the industries are learning more about their products, services and/or customers.

AI’s Role in Training

In a more simple role, it can be argued that data analytics has been used for years by the training industry. Current learning management systems and survey systems have the capacity to collect high level data analytics from learners. The issue has been how to objectively relate the data from the learners. 

We know that AI can assist with this from its use in other industries. AI can move the training team a step beyond to answering the ROI question effectively. To be able to answer this question appropriately, we need to be able to analyze behavior. This would include a more complex survey response and gathering data from outside the LMS. An example of this could be, if the training team desired ROI on customer service training, the AI would incorporate data from the customer satisfaction ratings, cost of training, retraining, etc.

AI, Learning Analytics & QAA

With AI, more data can be analyzed allowing for a complete learning analytics view. This will allow for organizations to be confident in their ROI answer. With known ROI, the future of training goals and objectives will be based on insights discovered within the data. 

QAA provides a customized Advanced AI Learning Analytics Tool (AAILAT) that aggregates data from multiple sources for analysis. Read our whitepaper or view the AAILAT demo to gain a better understanding of the customization potential. Contact us to schedule a time for a demo and answer questions.

As we work on a problem or in a crisis situation (i.e. COVID-19, project issues or delays, etc.), we may have difficulty thinking or planning ahead of the current situation. However, we need to understand that the problem will have a resolution which is determined by a timeline. As leaders, we need to prepare for the future after the situation has resolved. 

With current events, we have the potential to learn more than we ever have previously due to the amount of data available. What will you learn from all the data available?

Current Employee Trends – Spring 2020

Let’s consider the current trends. Who is working? Who is not working? Who is essential? Who is non-essential?

Looking at our stoplight image, it would seem that more employees are still considered essential and are still working on-site than not. However, as we look at images from traffic cameras and other global satellite images, we understand this is not necessarily true. The type of essential jobs may be greater, but these jobs may require fewer on-site people to complete in some instances. 

Some of the jobs are being completed in both locations. This is allowing for teams to rotate or for minimal to no public disruption of services. A final consideration are the employees that are completely remote. The majority of these employees were not remote before the current situation. After the situation resolves, how many employees can and will remain remote full or part-time?

Data Analytics & Learning Analytics for Lessons Learned

With this question in mind, no matter your area of industry, your next thought should be the potential lessons learned from data analytics and learning analytics.

As you begin to look at the data, here are a few questions.

1. Were your employees prepared with training to meet the challenges faced by this situation? 
2. Which training best prepared them? 
3. What training is lacking?
4. What training type of training method would be most effective for reach-back in this situation?

Employees will need a method to provide open-ended responses about the experience. You will need a method for capturing this data and analyzing it. Your leadership and the training team will need to determine the best strategy for gathering and analyzing the data for your business. The most important point is to capture and retain as much data as possible for the future lessons learned.

How QAA can Help

Our team can help by providing a customized Advanced AI Learning Analytics Tool (AAILAT) solution. AAILAT aggregates data from multiple sources for analysis. Our team assists in developing themes for the AI agents search for which leads to further insights from the data.

To learn more about the process, read our whitepaper. View our AAILAT demo to gain a better understanding of the customization potential. Contact us for more information about the customization.

You gather data from the learning management system (LMS), surveys, Human Resources and multiple other sources throughout the year. Why should you even consider purchasing or using an AI driven program for analyzing and summarizing data that you already have and know?

A Customizable Tool

Consider a Swiss army knife, it is not simply a knife. If someone asked you, what can you do with it? You would be able to provide the standard answers of corkscrew, can-opener, scissors, nail file, etc. However, the tools may be used for multiple uses. Our Advanced AI Learner Analytics Tool (AAILAT) provides an equivalent result for learning data analytics for organizations as the Swiss army knife does for individuals.

Customizable How?

AAILAT will gather data from multiple sources and perform qualitative and quantitative analysis for the team, saving time. What does this mean for you – learning and development managers and global training professionals? How can AAILAT be customized to fit my needs?

Similar to the Swiss army knife example, our team cannot provide a clear answer to this question. AAILAT’s customization possibilities is NOT limitless, but it is based on your learner questions and the types of learner programs from which data will be gathered. Typical programs include LMS, survey systems and others with reporting numerical data (i.e. HR, sales, etc.). Understanding what you want to learn from the data will help to determine how to use the tool.

Knowing When to Customize

Our team will help in the beginning by asking questions to establish the baseline of your customized AAILAT program. From here, updates will be made as needed based on the data gathered. To learn more about the process, read our whitepaper. 
Our AAILAT demo provides an idea of the customization available. Contact us and let us know what you think.

9 July 2019

A question asked of training and learning directors often is, ‘what is the return on investment for the training program?’ Then, the director must determine the best way to place value on the training. This usually involves review of the post training surveys, assessment scores, and costs associated with training. But, have you considered what other questions you might consider if possible?

Learning Analytics

First, consider what information you gather from surveys and assessments. 

• Pre and Post Training Surveys – Provide data on the learners (ex: demographics), opinions on the training (ex: methods, scenarios, etc.), and usability or relatability to the job. A survey may also be used to rate a trainer or facilitator to compare and improve instructor-led sessions.
• Assessments – Provide data on learner comprehension. By tracking the questions that are correct versus incorrectly answered, instructional designers can determine which content is structured appropriately. Also, if the same incorrect answer is chosen repeatedly, the instructional designer needs to review the content and the question. 

Beyond these data sets, what other information would you want to know to help improve your training program while proving the value? The answer depends on the type of organization and type of training provided. Organizations may desire details about training and employee safety attitudes or behaviors. Other organizations may want to search for any trends in training and learner job performance.

Take Away

Learning and training groups collect vast quantities of data with each new course or lesson. However, the more for more specific data involves collection beyond surveys and assessments. QAA uses multiple methods to gather data from all levels and views to achieve a full view of an organization’s training program’s value and needs.

We partner with Bintel, Inc to offer our customized Advanced AI Learning Analytics Tool. AAILAT allows training groups to drive into data to discover unknown insights because we capture, aggregate and conduct the analysis using an AI driven solution. Read our whitepaper to learn more.

Contact Us to discuss what analytics you want to discover in your organization.

22 April 2019

The title may seem funny to some, but we have been doing market research. (Note: it is not scientific in any way, other than we have a list of questions.) Every company, who has spoken with us, offers training to their employees. We spent approximately 10 to 15 minutes speaking to individuals within companies after assuring them that it was NOT a sales attempt. 

Our participants were from companies who employed 5 to more than 1000 people. The training consisted of simple on-the-job training with a mentor or was directed by a corporate office or vendor (COTS) training. Topics included on-boarding, health, safety, federal/state regulations, etc. All modalities were used from instructor-led to webinars with some companies using LMS and other tracking with spreadsheets. From our conversations, we were able to conclude that most companies do offer training to employees. However, these L&D programs do have areas for growth and improvement which were identified from our questions.

Established L&D Program – Why Learning Analytics?

If you have an established Learning and Development program that seems to be getting results, why would you need learning analytics? Good question. Here are some items that came up in our market research that should be considered.

• Training 
  • The courses you currently offer only provide minimal skill training for employees. For example: A forklift driver receives the minimal training to remain certified.
  • The courses do not allow employees the ability to cross train to other skill sets.  
  • The courses do not afford employees the ability to advance their skills in their current roles.
• Training Modalities
  • Some courses must be instructor-led. With increasing technology availability and advances, training could be offered using gamification, augmented reality, virtual reality, webinars, mobile and online modalities. 
• Training Tracking
  • L&D teams (or individuals) are required to track training completion. While the current spreadsheet or LMS may work, a new system may be desirable or cost effective.
• Overall Training Effectiveness
  • Some courses are required and mandated by regulations. Others have been in your companies system since start-up with little to no change. 
  • Are these courses effective? Do the courses meet the employee and company needs? Are employees following the procedures set in the training?

QAA’s learning analytics process assists with the identified items as described in the following table.

Learning Analytics – What’s the Cost?

QAA takes pride in working with our clients to provide a quality, premium, cost-effective service that is customized for each company. Our clients have different needs based on employee training levels and organization size. Therefore, our cost is determined by the estimated number of hours required to complete the training analytics process. As an additional incentive, we provide discounts to ensure we offer the most cost effective service. To learn more about the cost of Learner Analytics, view our video, ‘Think a Learner Analytics Solution Is Expensive‘.

Final Thoughts

We would like to thank the individuals and organizations who spent time answering our market research questions. Your answers were invaluable to our process.

You may watch our demo or read our whitepaper. To learn more, contact us.

6 February 2019

It’s a typical day at work – you walk into the breakroom and notice a pile of dirty dishes in the sink. Despite the sign, “Wash Your Dishes”, multiple email reminders, and discussions during staff meetings your co-workers have not changed their behavior.

Receiving a Training Certificate ≠ Success

Organizations can sometimes see the same results with training. Employees complete a training, but accidents and incidents still occur. Why does this happen? The employees passed the training assessments and received the certificate. Is there a way to measure effectiveness of the training?

In the following TedTalk, Bob Nease takes about 12 minutes to explain why we make lousy decisions. These bad decisions can be reflected back to our work behavior. We complete safety or cyber training. We understand that our company passwords should not be shared, be set at certain parameters, and never be written down. However, does that mean we always follow these rules? No, we need to repeat new tasks until they become learned behavior.

Summary

Behavior analysis through task analysis allows organizations to answer two important questions. (1) Are the employees following the skills learned in the training? (2) If not, what are the employees doing instead? As these questions are answered, the next steps can be determined to assist employees in making the tasks learned behaviors.

Contact us for how our AI Solution can assist in measuring the overall effectiveness of training through behavior and task analysis.