ESA Φ-lab have created an AI-powered digital assistant that allows users to access and explore complex Earth Observation data through a natural language interface.
“In the long term, the aim is to integrate this tool into digital twins of Earth, supporting decision-making in areas such as climate monitoring, disaster management and urban planning.
A digital twin of the Earth environment is an interactive “digital replica” that allows us to understand the various relationships between the physical and natural Earth environments and society. It enables scientists to quantify past, present and future events on our planet, integrating models, observations, and technologies such as Artificial Intelligence (AI) to improve our understanding of the human impact on global environment and society.
Through data and simulations, digital twins allow for real-time prediction, monitoring, control and optimisation of Earth’s natural and physical processes. Two related flagship programmes are the European Commission’s Destination Earth (DestinE) and the European Space Agency’s Digital Twin Earth (DTE).
ESA Φ-lab is funding the creation of a digital assistant interface for digital twins of Earth – Demonstrator Precursor Digital Assistant Interface for Digital Twin Earth (DA4DTE) – in collaboration with e-GEOS (Italy), the National and Kapodistrian University of Athens (Greece) and the Technical University of Berlin (Germany).
The developed digital assistant prototype interacts with users via two modalities, text and satellite images, and consists of four back-end engines: search-by-image, search-by-text, knowledge graph question answering (KGQA), and visual question answering (VQA) engines.
The Knowledge Graph Question-Answering Engine is built on a knowledge graph that integrates geospatial data from widely used geographical knowledge bases, such as OpenStreetMap, along with metadata from Copernicus missions. This knowledge graph is combined with Large Language Models (LLMs) such as Llama, GPT, and Mistral.
In the end, the idea is that users, whether they are EO experts or not, will be able to perform a semantic query on EO data archives such as “Show me 3 pictures of rivers in Italy, with a vegetation coverage over 20%, taken after May 2020” or “Count the number of buildings in this area”. This digital assistant will help to answer questions in several EO-related data domains – agriculture, forest, urban, marine, cryosphere, among others – contributing to improved decision-making.
By allowing users to ask questions in natural language and receive insightful, data-driven responses, this tool lowers the barrier to entry and accelerates the use of EO data for decision-making. Whether it is for climate monitoring, disaster response, or environmental management, having an intelligent interface to navigate and interpret vast datasets is essential for informed action.”
A Geodyssey - Geoscience Text Analytics and Enterprise Search Research 
Leave a comment