Abstract
Heritage Weaver investigates the use of multimodal AI to link and explore museum data across collections. Through a series of experiments, from zero-shot learning and information retrieval to record linking, we demonstrate the value of fine-tuning multimodal models on digital heritage. The paper elaborates on various evaluation strategies, leveraging existing metadata or using expert annotations, to measure improvements in the model’s “understanding” of often complex and messy historical materials.