Introduction:

When a patient with breast cancer begins chemotherapy before surgery, one of the most important unknowns is whether it will work. Patients who achieve what clinicians call pathological complete response, where no residual tumor is found after treatment, tend to have significantly better long term outcomes. Knowing this in advance would let doctors personalise treatment, reduce unnecessary side effects, and plan less invasive surgery. The challenge is making that prediction accurately before the surgery actually happens.

Most existing AI tools built for this task rely on MRI. That is a problem, because MRI is expensive, time consuming, and unavailable in many clinical settings, particularly in lower resource contexts. Mammograms, by contrast, are one of the most routinely used imaging tools in breast cancer care worldwide. Almost every patient already has them. Yet very little research has explored whether mammograms combined with the clinical data routinely collected during treatment could be enough to predict treatment response. That gap is what this project set out to address.

Working with post-chemotherapy mammograms and clinical records from patients at Queen Mary Hospital, Hong Kong, I developed a multimodal AI that fuses imaging and clinical information into a single prediction. The model uses Mammo-FM, a foundation model pre-trained specifically on mammographic images, to extract features from two standard views per patient. These image features are then conditioned on the patient's receptor subtype and combined with clinical variables including age and hormone receptor status. The goal was to let the clinical information guide how the model interprets the images, rather than treating the two data streams as entirely independent.

To measure how much each type of input actually contributes, we compared three model arms: one using clinical data only, one using imaging only, and the proposed multimodal model. The multimodal model consistently outperformed the imaging-only approach, and this improvement held within each molecular subtype separately, not just in the pooled results. A key finding was that mammograms alone did not simply re-encode which subtype a patient belonged to, confirming that the imaging contribution was genuinely independent of the clinical signal.

These results suggest that routine mammograms, when paired with standard clinical records, can contribute meaningfully to predicting treatment response without the need for MRI. This matters most in settings where MRI access is limited, which describes the majority of the world. Next steps include interpretability analysis to understand what the model is responding to in the images, and prospective validation to assess whether the findings hold in a real clinical setting.