Parameter efficient multi-model vision assistant for polymer solvation behaviour inference

Polymer–solvent systems exhibit complex solvation behaviours encompassing a diverse range of phenomena, including swelling, gelation, and dispersion. Accurate interpretation is often hindered by subjectivity, particularly in manual rapid screening assessments. While computer vision models hold significant promise to replace the reliance on human evaluation for inference, their adoption is limited by the lack of domain-specific datasets tailored, in our case, to polymer–solvent systems. To bridge this gap, we conducted extensive screenings of polymers with diverse physical and chemical properties across various solvents, capturing solvation characteristics through images, videos, and image–text captions. This dataset informed the development of a multi-model vision assistant, integrating computer vision and vision-language approaches to autonomously detect, infer, and contextualise polymer–solvent interactions. The system combines a 2D-CNN module for static solvation state classification, a hybrid 2D/3D-CNN module to capture temporal dynamics, and a BLIP-2-based contextualisation module to generate descriptive captions for solvation behaviours, including vial orientation, solvent discolouration, and polymer interaction states. Computationally efficient, this vision assistant provides an accurate, objective, and scalable solution in interpreting solvation behaviours, fit for autonomous platforms and high-throughput workflows in material discovery and analysis.