Federated Site-Invariant Adaptation for Medical Vision-Language Models Across Heterogeneous Radiology Networks

Abstract

Hospitals often differ in scanner vendors, acquisition conventions, report style, patient mix, and clinical language, causing medical vision-language systems to behave unevenly after transfer from one institution to another. Direct pooling of images and reports can reduce this problem, but pooling is frequently limited by governance rules, privacy constraints, data-use agreements, and institutional preferences. This study investigates whether federated site-invariant adaptation can improve cross-institution medical visual question answering without exchanging raw clinical images or reports. We built a six-site retrospective benchmark of 18,420 radiology image-question pairs spanning chest radiography, head computed tomography, abdominal computed tomography, and musculoskeletal radiography. A shared medical vision-language backbone was adapted using federated low-rank modules, site-adversarial representation alignment, and a prototype-based language harmonization loss. The method was compared with local-only tuning, centralized tuning, federated averaging, and a no-adaptation baseline. On held-out external-site testing, the proposed method improved macro accuracy from 63.8% to 72.4%, reduced site disparity from 14.9 to 6.2 percentage points, and improved worst-site calibration error from 0.181 to 0.084. A mixed-effects analysis showed significant gains in scanner-shifted computed tomography cases and in report-style-shifted chest radiograph questions. Privacy-preserving gradient audits indicated low memorization risk relative to centralized tuning. The results suggest that federated adaptation can reduce institutional brittleness in medical vision-language systems while preserving a degree of local data control, although residual errors in rare findings and small sites remain important