AI Tutor for University Admission Exam
Using NLP, Tensorway created an advanced system able to give students detailed human-like feedback on their essays.
analysis
Story behind
Our client, a dedicated tutor, teaches students how to write essays for the Graduate Australian Medical School Admissions Test (GAMSAT). Recognizing the importance of high-quality essay feedback, the client wanted to develop an online automated essay feedback system.
The existing system by ACER (organizer of the exam) was outdated and could only provide approximate grades. Our client believed this system could be easily outplayed with the latest AI advancements.
Frasers, another GAMSAT preparation service, has announced its new AI product, also aiming at helping students improve their essay-writing skills.
Our client approached Tensorway, known for its expertise in developing customized AI models, to create a competitive solution that could save time and provide detailed feedback to his students.
Goal
The primary goal was to develop an advanced, accurate, and efficient automated feedback system tailored to the GAMSAT essays. The system needed to evaluate various aspects of the essays, including:
Additionally, the system aimed to provide detailed feedback and potentially include functionalities like a plagiarism checker.
Model's feedback
"This essay demonstrates a solid understanding of the ethical considerations surrounding the integration of AI in healthcare. The structure is clear, and the points are well-articulated, particularly the emphasis on patient privacy, data security, and bias in AI algorithms. However, the discussion on the doctor-patient relationship could be expanded to explore more nuanced implications and potential solutions..."
Challenges
Data collection
Gathering a substantial and representative dataset was a significant challenge. Achieving accurate results required a minimum of 500 high-quality essay samples with corresponding grades and feedback, which necessitated extensive coordination and effort.
Model selection and training
Identifying and fine-tuning a base large language model (LLM) that could effectively handle the nuanced and specific requirements of GAMSAT essays posed a considerable challenge. The process required deep expertise in natural language processing (NLP) to ensure the model's performance met the high standards needed for accurate feedback.
Setting up instructions and RAG pipeline
We boosted the model performance by setting up additional sources of information that it could use to provide better answers. We aggregated data from different sources - tutor’s books, courses, and educational videos. Then, we created additional instructions on what errors to look at, providing models with more examples of how to handle different cases.
As a cherry on top, we taught the model to write essays not only in the style of the tutor but also to identify nontrivial aspects of essays, making it as good as our client in identifying weak spots in essays and suggesting how to improve them.
Integration
Seamlessly integrating the advanced AI model into an accessible and user-friendly online platform was a complex task. This required sophisticated software engineering to ensure that the system was both robust and intuitive for students to use without unnecessary effort.
Ensuring quality feedback
Providing feedback that matched the level of expertise of our client was a demanding challenge. The model needed to generate nuanced, human-like feedback that was both detailed and educational, requiring careful design and rigorous testing to meet the high expectations of academic quality and instructional value.
Tensorway’s solution
Tensorway proposed a comprehensive approach involving both AI model development and robust software integration to create an automated essay feedback system that functions as a high-level AI tutor with a large language model (LLM) underneath.
AI model development
Data preparationand model training
Collected and prepared a large selection of essays with corresponding grades from our client’s students.
Selected a suitable pre-trained open-source model and fine-tuned it using the specific essay data.
Trained the model on the collected data, employing retrieval-augmented generation (RAG) models for data augmentation, enriching essays, and improving context understanding.
Feedback generation
Used a trained model to provide detailed feedback on essays.
Designed instructions based on our client’s criteria to ensure relevant feedback.
Ensured the feedback resembled our client’s writing style, identified errors, and provided valuable recommendations.
Software development
Frontend and backend development
Designed an interface for essay submission where students can specify the topic, time spent writing, and the essay text.
Integrated Google authentication.
Built a secure database to store essays, grades, and user information.
Integrated a credit system where students pay to get credits for essay checks.
Integrated the system with SamCart to ensure secure credit transactions and fixed potential fraud issues.
Testing and deployment
Conducted internal testing.
Configured the production server and deployed the application.
As a result...
The AI tutor allowed our client to process more student work at a time, making GAMSAT preparation more efficient. Thanks to the AI tutor, the client could focus on other educational projects and more guided mentoring of his students.
Tensorway’s model provided detailed feedback on various essay metrics such as overall quality, structure, clarity, and specific sections. This helped students understand their strengths and areas for improvement more clearly, ultimately enhancing their essay-writing skills. Notably, the feedback was so human-like that the client was amazed at how closely it resembled his own writing style.
The solution allows students not just to receive feedback on one essay, but iteratively improve the essay and evaluate it automatically, keeping the tutor away from the process. It is also cheaper and makes education more accessible.
I’m glad we ended up getting a working model of the project live. It’s significantly better than Fraser’s AI and I’ve had many students say they were amazed by how comprehensive it was and similar to what I actually said to them about the same piece of writing. Thanks for all your help and the team’s hard work.
Project team, steps, and timeline
Team
Data preparation and model selection
Model training and initial testing
Creating RAG pipeline, instructions, and further model tuning
Final testing, integration, and deployment
Web development
Other possible
applications
The developed automated essay feedback system can be adapted for other standardized tests and academic purposes, such as:
Other admission exams
Similar automated feedback systems can be developed for different entrance exams.
Education institutions
Schools and universities can use the system to provide detailed essay feedback to students.
Corporate training programs
Companies can use the model to evaluate and improve the writing skills of their employees.
Tutors
aiming at optimizing their work with AI.