Network of dots - photo by Cash Macanaya from Unsplash
Researchers at the University of Sussex Business School have designed, led and completed the judging of a major global innovation challenge that could change how innovative research is identified, evaluated, and supported.
The Metascience Novelty Indicators Challenge, co-hosted by the Business School’s Science Policy Research Unit (SPRU) alongside the UKRI Metascience Unit, Coefficient Giving, RAND Europe, and Challenge Works, brought together 30 research teams from around the world to develop tools capable of identifying novelty in scientific research.
AI and the challenge of measuring research novelty
The Challenge addressed a longstanding problem in how research is assessed. While novelty and innovation are widely regarded as the height of excellence in science, they can be very difficult to measure. As a result, significant discoveries may take years or even decades to be recognised through traditional means such as citations.
To tackle this gap, researchers from SPRU and RAND Europe created one of the largest datasets ever gathered on research novelty. Over 41,000 experts assessed the novelty, uniqueness, and innovativeness of 37,480 recent academic papers from across all disciplines. The organisers then used their assessments to create a benchmark dataset for comparing AI-based innovation tools and competing datasets.
Building a benchmark based on expert judgement
The organisers did not decide in advance what counted as novel research. Instead, they based the benchmark on the combined judgements of the experts. This allowed for different kinds of novelty, including new theories, methods, datasets, discoveries, and new combinations of existing ideas.
Thirty international teams each developed indicators to predict how experts would rate the novelty of the same papers. Their predictions were then compared with the benchmark dataset using several statistical tests to check how accurate and consistent they were.
LENS approach comes closest to expert assessments
The winning team - focusing on AI for energy research, from the Jülich Research Centre, Germany, developed an AI-powered system called LLM-Evaluated Novelty and Significance (LENS). It checks the full texts and references of research papers to assess how much new ideas and knowledge they contribute to science. The team will receive a £300,000 grant to further develop the tool.
Early findings show that the LENS approach came closest to matching how experts first judged research novelty. It implies that AI systems can be trusted to detect patterns of novelty and innovation in research even when people have different opinions.
Dr Sarah Otner, Principal Investigator, project lead and designer, and researcher at SPRU, said:
“We were not seeking perfection in our approach to the Challenge; we were looking for good modelling of what (statistically) noisy, disagreeable humans do. SPRU, RAND Europe, and Challenge Works are delighted to have delivered this project, and we are excited to work with the Challenge teams and the whole metascience community to continue exploring these research tools and their applications.”
Commenting further on the Challenge Prize, Ben Steyn, Co-Head of the UKRI Metascience Unit, said:
“In a new academic world of AI and human slop, novelty indicators can help shine an early light on research that might be important, without having to wait decades for ‘sleeping beauties’ to be discovered (or not). This result shows that, while there is disagreement, there is also some significant consensus, and that consensus can be detected computationally more reliably than a given human can detect it.”
What this means for research assessment
The competition has attracted international attention, including coverage on the Science platform. It described the project as a new attempt to put into numbers one of science's most important but “notoriously thorny to define or measure” concepts.
The Challenge was launched in September 2025 and represents more than 18 months of collaborative work. The SPRU team designed and delivered the study that centred on the judgement of human experts as the ultimate decider in the competition, informing the benchmark dataset used in evaluating the AI tools of competitors.
With data collection now complete, researchers at SPRU and RAND Europe will further analyse the findings. They will also try to understand how experts think of novelty and produce peer-reviewed research papers. The team also plans to develop a toolkit for research funders, evaluators, and policymakers interested in applying the indicators and learning in real-world decision-making.
SPRU’s role in advancing metascience
This is the first project from The Metascience Research Centre based in SPRU. It is part of the Business School’s growing contribution to the field of metascience, which is the study of how research itself is conducted, evaluated, and improved. It highlights our role in shaping future approaches to research assessment worldwide.
Study at SPRU
In 2026, SPRU celebrates 60 years of world-leading science policy research and education. Find out more about post-graduate study at SPRU: Masters and PHD.