RockGPT – Impact of Retrieval Augmented Generation on retrieval and reasoning.

Interesting paper on the impact of knowledge curation, rather than large language models, on outcomes for geoscience. Self-hosted code is open-source. From the models, themselves, Phi-4 and Llama are best performing. Bailo et al (2026) link in the comments.

Abstract
Large Language Models (LLMs) are increasingly adopted in scientific research, yet their use raises important challenges related to transparency, provenance, reproducibility, and scientific knowledge governance. In the geosciences, where conclusions often depend on integrating multidisciplinary evidence, ensuring that AI-generated responses can be traced to verifiable scientific literature is particularly important. This paper presents and evaluates a methodology for trustworthy scientific knowledge management that places researcher-curated knowledge governance at its core. The approach is based on four complementary principles: researcher-curated knowledge, retrieval-based grounding, traceability and provenance, and institutional governance. The methodology was implemented through RockGPT, a self-hosted geoscience-oriented AI system combining open-source LLMs with Retrieval-Augmented Generation (RAG) and a curated corpus of scientific publications on Italian seismotectonics and the Central Apennines. The system integrates Ollama, OpenWebUI, and six open-source LLM families (Qwen, LLaMA, DeepSeek, Mixtral, Gemma, and Phi). Evaluation combined a comparative quantitative assessment with a qualitative governance-oriented analysis. Across all evaluated models, access to the researcher-curated corpus consistently improved response quality, with the largest gains observed in tasks requiring synthesis across multiple documents. Beyond these quantitative improvements, the qualitative analysis showed that the proposed approach supports key requirements for scientific AI, including transparency, provenance, reproducibility, data sovereignty, and human oversight. The results indicate that reliable scientific AI depends not only on model capabilities but also on how scientific knowledge is selected, curated, governed, and connected to generated responses. The proposed methodology demonstrates how researcher-curated knowledge governance can provide a foundation for transparent, reproducible, and institutionally controlled AI systems for scientific knowledge management in the geosciences and beyond.

https://link.springer.com/article/10.1007/s12145-026-02197-5