First peer reviewed research paper on Geological Aware Large Language Model (LLM) RAG system

A Geological Aware Large Language Model to be released November 14th 2024: University researchers have experimented using open-access geoscience papers to guide OpenAI’s GPT-4, a system they call “GeologyOracle”.

The authors, Baucon and Neto de Carvalho, are from the University of Genova and Lisbon respectively, the latter also from UNESCO Global Geoparks. As the paper was submitted in February 2024 it is believed this is the first peer reviewed research paper on such a geoscience LLM system.

The broad sources are listed but not specific journals or volumes of papers. Although not specifically stated, it appears (?) the researchers have used a Retrieval Augmented Generation (RAG) arrangement, rather than attempting to fine tune GPT-4 further.

“A total of 152 questions were proposed by a diverse panel of 24 geoscience experts, who critically evaluated the answers provided by the AI on a 0–10 scale. The central tendency measures, including the mean score of 6.5 and the median of 7.” Results showed 80% of answers were deemed a pass.

The authors then discuss other possible use cases for Geoscience including the newer GPT-4o version.

It is a very interesting published research paper well worth a read.

I would like to have seen a baseline comparison against the 152 questions to GPT-4o as it generically is, without the use of geoscience specific RAG. This would support the need to further guide such LLM’s. Publishing the full list of open-access papers used in RAG, would also support Open science and aid transparency as well as the 152 questions benchmark. Apologies if I have missed these, I did not see them in the paper.

Hallucinations and errors are a concern as with all LLM’s. The use case of GeologyOracle to support Geotourism is an interesting idea when image support is available. There may be opportunities to support outreach, academia, mining exploration, critical minerals, geothermal, geotechnical, ground investigation, hydrogeology, geohazards, natural hydrogen, oil and gas exploration etc.

It appears “GeologyOracle” will go live on November 14th 2024, no other details are known at this time. I look forward to finding out more.

Paper: https://link.springer.com/content/pdf/10.1007/s12371-024-01011-2.pdf

Link to GeologyOracle: https://geologyoracle.com