Events

We will discuss two new frontiers in science.

1) Autonomous AI Scientists: Research conducted by autonomous AI scientist systems is rapidly increasing in prevalence. These systems execute the entire research process autonomously, with little or no human intervention. While papers generated may appear appealing, we investigate whether these systems follow rigorous research methodologies. Using novel experiments designed to mitigate confounding factors, we uncover significant and concerning methodological flaws in their workflows. We also propose a method to detect these problems and provide policy recommendations for journals and conferences: Such methodological problems are not detectable from the produced paper alone but can be identified through analysis of the trace logs of the workflow executed by the AI scientists.

2) Lotteries for Acceptance Decisions: Traditional decisions for accepting/rejecting papers or grant proposals involve expert reviews followed by discussions and human-specified decisions. More recently though, citing drawbacks of such traditional approaches, a number of funding agencies worldwide have moved towards a different decision model to make acceptance decisions. These agencies have incorporated “partial lotteries” into their decision-making, where final decisions are randomized in a manner that still respects reviewers’ evaluations. We will first identify several problems in current implementations of such partial lotteries. We will then present a principled approach to designing improved partial lotteries with strong mathematical guarantees and empirical performance.

The talk will also contain a generous dose of minions.