TeacherMatic Further Education pilot report
The TeacherMatic pilot report details a collaborative initiative by Jisc’s AI team and TeacherMatic, aimed at reducing educators’ workload and enhancing productivity through AI-assisted tools, with positive feedback on its ease of use and time-saving benefits for further education.
Contents
This report provides a comprehensive review of TeacherMatic, detailing its initial pilot implementation, evaluation results, participant experiences, future development plans, and a summary of key findings
Introduction
Piloting TeacherMatic
What is TeacherMatic?
Piloting institutions
The pilot process
The pilot was divided into two distinct phases: the first phase spanned from May 2023 to June 2023, while the second phase extended from June 2023 to February 2024. Each participating institution received 50 licenses for 12 months, enabling around 400 participants to actively engage with and evaluate the
TeacherMatic platform.
Participants attended a training session designed to ensure they were able to use the tool. This session was a joint effort, collaboratively conducted by the TeacherMatic team and the Jisc team. The goal was to equip all participants with a thorough understanding of the platform’s functionalities and features, enabling
them to use TeacherMatic effectively.
Phase review process
At the end of phase one, five to ten participants from each institution took part in a focus group meeting. These meetings, held in June and July 2023, provided a forum for participants to share their experiences and insights. Then, in October 2023, all participants were invited to complete a mid-phase questionnaire. This survey aimed to gather opinions on the ease of use of the platform, identify the most frequently used features, and evaluate functions available in content generation.
In February 2024, all participants were asked to complete a second questionnaire. This survey was designed to evaluate the progress and effectiveness of the TeacherMatic platform since the beginning of the pilot. It covered participant interaction, including the frequency of use, the type and quality of content created, as well as the experiences with new features added during the pilot. The aim was to gather in depth feedback on the platform’s development and how well it met participant’s needs throughout the pilot
period.
The evaluation of phase two also included a second component: individual review meetings with the pilot lead from each institution. These sessions were designed to discuss the overall impact of the pilot and examine the processes in depth, including onboarding, staff training, and reflections on the pilot
TeacherMatic pilot evaluation
Findings from phase one
Perspectives from focus groups
Throughout the focus groups, all eight of the colleges were positive about TeacherMatic. We discussed usability, the speed of content generation compared to traditional methods, the quality of resources generated, and the time saved by using the AI tool.
In summary, feedback covered:
Positives:
- Easy to use interface and simple to generate content
- Useful blue icons and videos explaining how to use generators
- Potential to help workload
- Step by step generator was excellent
- The sliders and Bloom’s taxonomy of learning features are useful when generating content
Suggested areas of improvement:
- It would be useful to be able to store query history
- Multiple generators in one could be useful i.e. Scheme of work and lesson plan
- Multiple-choice question generator sometimes does not produce the answer
- Chat generator could be slow
- Add the duration of the lesson to the scheme of work generator
Ideas for new generators to add:
- A presentation generator
- To be able to share created content with peers directly within TeacherMatic
Response to phase one feedback from TeacherMatic
After receiving feedback from the first phase of the pilot, the TeacherMatic team responded swiftly. Specific changes were implemented in response to participant suggestions, enhancing both functionality and user experience.
Implemented updates:
- ‘It would be useful to be able to store query history’: this has now been added and users can save results
- Multiple-choice question generator sometimes does not produce the answer’: this has been resolved
- ‘Chat generator could be slow’: this has now greatly sped up
- A presentation generator has been added
- ‘To be able to share created content with peers directly within TeacherMatic’: this has been added so users can share content
Findings from phase two
Perspectives from mid-pilot questionnaire
In October 2023 we asked participants to complete a mid-phase questionnaire, this was due to the development and additions to TeacherMatic partly based on the feedback from phase one. Out of 400 participants we received 102 responses. A key point to raise at this juncture would be the possibility of
survivorship bias, in that the results may show a potential bias towards those that continued to use the tool and provided feedback against those that did not. This is something to keep in mind when considering the following.
In Summary:
- Feedback showed that 99% of respondents were able to create content effectively
- Participants were asked how often they use TeacherMatic; 53% weekly, 36% monthly, 3% daily, and a small group had not tried it yet
- The most used generators were lesson plans, multiple-choice questions, scheme of work, PowerPoint, and classroom questions
- The quality and accuracy scores were rated at 4 out of 5
- The features like complexity levels and learning needs were rated 3.5 out of 5
- Participants rated Bloom’s taxonomy function at 3.6 out of 5
- 87% of the respondents reported that TeacherMatic has saved them time, while 13% indicated that it had not
- 93% of respondents stated they would continue using the tool, while 7% said they would not
- Participants were asked if TeacherMatic has had a positive impact on learner experience; 13% said very positive, 53% said positive, with 1% responding negative
- 76% of respondents would recommend TeacherMatic, while 24% indicated they would not
- A presentation generator has been added; however, feedback showed that it can generate irrelevant images and generic text
Perspectives from phase two participant questionnaire
At the end of phase two, all participants were asked to complete a questionnaire aimed to review the usability, and quality of content and to gather how much time can be saved utilising TeacherMatic over a longer period. Out of 400 participants we received 179 responses. As above, in respect of the possibility of
survivorship bias, it is worth noting we received an additional 77 responses during phase two.
In Summary:
- Participants were asked if TeacherMatic has saved them time and 79% said yes
- On average users said they saved two hours ten minutes per week by utilising TeacherMatic
- Participants were asked to rate their experience using TeacherMatic for content generation in terms of ease of use; 28% said excellent, 51% said good and 21% said average
- Feedback showed when asked about how participants would rate the accuracy of the content generated by TeacherMatic within a specific topic or subject; 16% very accurate, 69% somewhat accurate, 11% neither accurate nor inaccurate, and 4% said somewhat inaccurate
Feedback in Summary:
- Feedback shows that when asked to assess the overall quality of the content produced by TeacherMatic; 19% said very good, 51% good, 27% fair and 3% said poor
- On a sliding scale of 1 to 5 stars participants were asked to rate the complexity slider based on the content generated when used, the rating average was 3.7 with 52% rating 4 stars
- On a sliding scale of 1 to 5 stars participants were asked to rate the learner needs function on the
content suitability generated when used, the rating average was 3.8 with 60% rating 4 stars - On a sliding scale of 1 to 5 stars participants were asked to rate the Bloom’s Taxonomy function on the content generated when used, the rating average was 3.8 with 40% rating 4 stars
Feedback in Summary:
- Participants were asked which method they typically use to input information to generate content through TeacherMatic; 55% said by using keywords, 7% by using the upload a file function, 1% use the URL function, and 37% use all three. 70% of participants have used the generator filter by sector
- 79% of participants said that they use the filter function to search for generators by sector, however, it was noted that there are now so many generators it is important to use the favourite function on the dashboard or else it can’t take time to find relevant generators
- On average, participants gave a rating of 7.7 out of 10 on of a scale of how likely they would be to recommend TeacherMatic
- Participants were asked if the pilot has helped to enhance their understanding of generative AI and its capabilitie;. 43% strongly agreed, 43% agreed, 11% were neutral, 2% disagreed and 1% strongly disagreed
- 69% of participants said that TeacherMatic has helped improve learning experiences. Participants said it has helped add variety to teaching activities and delivery methods (resource ideas, quizzes, plenary activities, lesson planning and activity ideas, differentiation of learning)
Perspectives from pilot meetings
At the end of phase two in February 2024, we held meetings with each pilot lead to discuss their experiences and gather feedback. Our discussion began with the topic of workload reduction. All leaders agreed that TeacherMatic can help to reduce the workload for some administrative tasks and resource creation. Although they noted some important points. For instance, to fully benefit from the platform’s features, users needed to dedicate some time initially to developing content by trialling different generators and different keywords. Furthermore, it was identified that to maximise the benefits of TeacherMatic, actively sharing ideas and collaborating in content generation workshops and sessions is essential.
In our review, we looked at the onboarding process, which was generally regarded as straightforward since users could easily create an account. However, there was a suggestion that it would be beneficial for the pilot leads to have visibility into which users were actively using the tool. We also discussed how TeacherMatic was received across the institutions. The feedback was largely positive, with the majority highlighting the platform’s time-saving benefits. It was particularly noted that TeacherMatic is a valuable resource for staff new to teaching and for adapting to changes in course specifications.
Continuing our discussions, we addressed the topic of AI literacy. The consensus was that for staff with little to no understanding of generative AI, TeacherMatic has been instrumental in illustrating how AI can be a valuable tool. However, some participants fed back that they can get more innovative and engaging content from other AI chatbots when users have good prompting skills. Additionally, it was noted that some staff members had prior experience with generative AI chatbots and were proficient in creating content for teaching and learning even before the pilot began. Lastly, when discussing the cost of TeacherMatic, most participants felt that the pricing was in line with their expectations.
Participant’s experiences with TeacherMatic
Strengths of TeacherMatic
All the colleges concurred that TeacherMatic was both quick and easy to use, especially highlighting its user-friendliness for those new to AI tools. They praised the platform for its ease of use, and the results when generating content for general topics.
The quality of content produced by TeacherMatic is generally high and appropriate for the intended audience. It was noted that the prompt provided by users impacts the specificity of the generated output.
Additionally, the ability to export content in a variety of file types is good.
Areas of development
Some participants found that the learning activities generated by TeacherMatic can be repetitive.
The difficulty level of content produced by the generators is sometimes considered not challenging enough for certain more detailed subject areas.
The quality of the presentation generator is not as good as other AI tools, with large amounts of text and irrelevant images. Some participants mentioned accessibility needs when generating presentations the presentation did not generate with suitable text formats for screen readers automatically.
Some participants said TeacherMatic is easy to use, however, with the number of generators, it can be difficult to explore all generators on one screen. That said there is a favourite function, and users can search for generators by sector.
A few pilot leads suggested a dashboard feature showing individual user activity for better tracking and support.
The time-saving aspect of TeacherMatic was praised and the tool has helped to add variety and creativity within teaching and learning.
The pace of development has been rapid, now offering over 50 different generators. These have been designed to ease the workload across teaching, management, administration, and support roles.
The TeacherMatic team was proactive in responding to feedback to strengthen the platform.
TeacherMatic: the team
All colleges were very positive about the TeacherMatic team, who had provided a wealth of experience and understanding about the further education sector. The team was very quick to respond to feedback and make improvements to the platform. The initial training was praised, and the support received throughout
the pilot. TeacherMatic has built a community for its customers which looks at how to get the most from the AI tool and to share experiences, this has been well received.
Wider perspectives of the pilot
Colleges expressed the following benefits of participating in the pilot:
- The pilot effectively enhanced AI literacy among staff, notably increasing their understanding of
generative AI capabilities - How generative AI can help with staff workload and wellbeing
- The potential to improve learner experiences with added variety in teaching, learning and assessment
TeacherMatic’s roadmap
Since evaluating the pilot, more generators have been added with now over 70 generators. These include additions such as “Ask Someone Famous”, “Survey Creator”, “Start of Lesson Recap”, and “Likert Survey Creator”, among many others designed to support marketing and administrative tasks.
There has been an addition to the Multiple-Choice Questions Generator which now allows users to integrate
YouTube videos directly. By copying and pasting a video link, the tool can now generate multiple-choice questions and answers based on the video content. Additionally, Microsoft Single Sign-On (SSO) has been implemented to streamline the login process.
Looking ahead there will be the introduction of a mobile app and meta generators which will combine the outputs of a number of generators from a single prompt.
Summary
The pilot indicates that TeacherMatic is well-suited for further education and can save users’ time. Participants commended the platform for its ease of use, efficient content generation, and benefits to workload. Feedback also highlighted areas for improvement and new feature suggestions which the TeacherMatic team were very quick to take on board and where possible, implement.
Participants found TeacherMatic to be user-friendly, particularly praising its easy-to-use interface and simple content generation process. The platform was noted for its instructional icons, videos, and features such as Bloom’s taxonomy, which assists in creating educational content efficiently. However, suggestions
for enhancements include the ability to integrate multiple functionalities into single generators. It also remains essential for users to evaluate the generated content, ensuring it is suitable and accessible to the
intended audience.
This is a great product. I have been experimenting with this and chat GPT, it's even come up with topics I had missed out in the curriculum and ideas for active learning I had not considered.
TeacherMatic was well-received across institutions, for its capabilities, especially beneficial for new teaching staff and those adapting to changing course specifications. It also improved AI literacy among staff, particularly valuable for those previously unfamiliar with generative AI, while those with prior experience found it useful for creating teaching content. Pricing was generally seen as reasonable, aligning with most participants’ expectations. TeacherMatic has been well-received, with a majority of participants recognising its benefits and expressing a willingness to continue using and recommending the tool.
