Optimizing zero-shot text-based segmentation of remote sensing imagery using SAM and Grounding DINO

The use of AI technologies in remote sensing (RS) tasks has been the focus of many individuals in both the professional and academic domains. Having more accessible interfaces and tools that allow people of little or no experience to intuitively interact with RS data of multiple formats is a potential provided by this integration. However, the use of AI and AI agents to help automate RS-related tasks is still in its infancy stage, with some frameworks and interfaces built on top of well-known vision language models (VLM) such as GPT-4, segment anything model (SAM), and grounding DINO. These tools do promise and draw guidelines on the potentials and limitations of existing solutions concerning the use of said models. In this work, the state of the art AI foundation models (FM) are reviewed and used in a multi-modal manner to ingest RS imagery input and perform zero-shot object detection using natural language. The natural language input is then used to define the classes or labels the model should look for, then, both inputs are fed to the pipeline. The pipeline presented in this work makes up for the shortcomings of the general knowledge FMs by stacking pre-processing and post-processing applications on top of the FMs; these applications include tiling to produce uniform patches of the original image for faster detection, outlier rejection of redundant bounding boxes using statistical and machine learning methods. The pipeline was tested with UAV, aerial and satellite images taken over multiple areas. The accuracy for the semantic segmentation showed improvement from the original 64% to approximately 80%–99% by utilizing the pipeline and techniques proposed in this work. GitHub Repository: MohanadDiab/LangRS.