Client Overview
The client is a medical education technology company based in Massachusetts, United States, that provides online learning tools to radiology residency programs, fellowships, and teaching hospitals. Its library contains thousands of imaging cases across X-ray, CT, MRI, and ultrasound, and its platform is used by program directors, faculty, and residents across the country. The company wanted to move from a static case library to an intelligent training platform that could automatically evaluate trainees at scale.
Project Summary: Radiology residents learn by reviewing many cases, but feedback usually depends on faculty who have little time to grade each one. We developed an AI-powered training platform that scores how trainees mark findings on images, compares their reports with expert references, and returns detailed feedback within seconds. Program directors get clear progress analytics for every resident. The challenge was making automated scoring fair, consistent, and trusted by experienced radiologists.
Key Challenges
Feedback was the biggest bottleneck. Faculty reviewed trainee work by hand, often days after a case was completed, and could only comment on a small sample of cases. Residents practiced a lot but learned slowly because they rarely knew right away whether they had found the right abnormality or missed a subtle one.
Grading was also inconsistent. Two faculty members could score the same answer differently, depending on experience and time pressure. Programs had no objective way to compare residents fairly across rotations, sites, or years, making it harder to identify who needed extra support.
The platform’s existing quizzes were limited to multiple-choice questions. They tested recall but not the actual skill of reading an image, such as locating a small nodule, outlining a lesion, or describing a finding in the right terms. The client wanted trainees to practice the real task, with answers that could be scored accurately.
Case content added further work. Building a good teaching case meant collecting images, expert annotations, and model answers, all carefully de-identified. Preparing each case took many hours, so the library grew slowly, and many topics had too few examples.
Finally, any solution had to work as a multi-tenant SaaS product for many institutions at once. It needed to handle large imaging files quickly, protect image data through strict de-identification and HIPAA-aligned safeguards, meet university security reviews, and fit into existing learning management systems and single sign-on. Radiology faculty also had to trust the scores before they would accept them.
Our Solution
We began with workshops involving the client’s product team, practicing radiologists, residency program directors, and engineers. Together we defined what a good answer looks like for each case type, which skills the platform should measure, and how scores should map to learning goals. Faculty also set the rules for what the AI would score automatically and what would still go to human review.
Next, we built the case and image foundation. A de-identification pipeline strips patient details from imaging files and metadata, and each case is stored with expert-drawn reference annotations, accepted findings, and a model report. Faculty can create new cases through a simple authoring tool, which shortens the time needed to grow the library.
At the core is a computer vision scoring engine. When a resident outlines or marks a finding, the engine compares it with the expert reference using measures such as overlap, location accuracy, and boundary precision. It identifies correct findings, partial matches, false marks, and missed abnormalities, and turns them into a score that is the same every time for the same answer.
For written work, we added a natural language module that reads the resident’s report or impression and checks it against the expected findings, terminology, and diagnosis. It recognizes different ways of describing the same finding, flags missing key details, and gives short, specific comments, such as noting that a lesion’s location was described but its size was left out.
To make feedback useful for learning, each result appears immediately with the expert answer overlaid on the trainee’s markings, a breakdown of what was found and missed, and a plain-language explanation. The platform tracks each resident’s performance by modality and body system, and then recommends the next cases to practice, starting with the areas where the resident is weakest.
The platform is a multi-tenant SaaS product, so each institution has its own secure workspace, branding, and user roles. It connects to learning management systems and single sign-on, and runs on elastic cloud infrastructure that handles large image files and many simultaneous users, such as before board exams.
Alongside the scoring engine, we built a program dashboard so administrators and faculty can manage training without technical support. Configurable views and settings were created for elements such as:
- Case library and assignment tools
- Scoring rules and passing thresholds
- Resident progress and milestone reports
- Faculty review and score override queue
This gave program directors a clear view of each resident’s development and let faculty step in whenever they wanted to adjust a score.
Before launch, the scoring engine was checked against expert grading on thousands of past resident answers. Radiologists reviewed the cases where the AI and human scores differed, and thresholds were tuned until agreement met the agreed target. The platform was then piloted with a few residency programs and released to more institutions in stages, with encryption, access controls, and security reviews completed at each step.
This tailored development is part of our broader AI work, where we build custom SaaS and computer vision solutions for healthcare and education companies that need accuracy, scalability, and trust from the start.
Technologies We Used
Key Results
The platform was piloted with residency programs and compared with the client’s earlier case library and manual grading process. The outcome was faster feedback, fairer scoring, and a training experience that residents used far more often.
- Feedback time dropped from several days to under 10 seconds per case.
- Automated scores matched expert radiologist grading with 92% agreement, compared with 84% agreement between individual faculty graders.
- Faculty grading workload fell by 70%, freeing time for teaching and complex discussions.
- Resident case completion increased by 58%, since trainees could practice and get feedback at any time.
- New case creation time fell by 45% with the authoring and de-identification tools.
- The client onboarded three times as many institutions within the first year, with zero critical findings in university security reviews.
Words of Appreciation
“Our residents finally get meaningful feedback the moment they finish a case, and our faculty trust the scores because they can see exactly how they were calculated. It has made training more consistent and given us a much clearer view of how each resident is progressing.”
Future Scalability
The modular design lets the client add new specialties, such as pathology, dermatology, or emergency medicine, without rebuilding the platform. The scoring engine can learn from faculty overrides and new expert annotations, so it improves as more cases are added. As more institutions join, the multi-tenant cloud setup can scale with them, and the same foundation can support simulated board exams, competency tracking, and continuing education for practicing radiologists.
