Should Dinghui lower the acceptance threshold? Use game theory to explore optimal review and decision-making mechanisms

In recent years, the field of artificial intelligence has increasingly criticized the review mechanism of large-scale computer conferences. The contradiction behind all this stems from the inconsistent interests of paper authors, conference organizers and reviewers:

• Paper authors hope that their papers will be accepted by conferences;
• Conference organizers hope to receive more high-quality papers to improve the reputation of the conference (conference Quality);
• Reviewers want to avoid excessive review workload (review pressure).

Therefore, how to balance conference quality and review pressure in an environment where the number of papers has increased significantly is the core issue to achieve a balance of interests among the three parties. Last year, scholars from the field of artificial intelligence put forward numerous opinions and suggestions on how to improve conference review and decision-making mechanisms. These ideas were summarized in a 23-page Google document. One of the ideas is very interesting and has been recognized by many people:

##Document link: https: //docs.google.com/document/d/1j7Mn2ZkquSzWJ_EzxdXBP3z_JQtrSeUa-CQ0gotAuYw/mobilebasic

This idea stems from such a counter-intuitive phenomenon, which this article calls reinvestment Paradox (resubmission paradox):

A large number of papers will be rejected every year (the acceptance rate of top artificial intelligence conferences such as NeurIPS is often less than 30% all year round), and most of these papers will be rejected in only Participating in re-submissions with minor adjustments or even no changes at all will always be accepted by the same conference or conference at the same level. Since most papers will eventually be accepted, why not lower the acceptance threshold so that more papers can be accepted after fewer resubmissions? This will prevent the same paper from being read repeatedly by reviewers and reduce review pressure.

Although this idea seems very reasonable, the author of this article proposes to use a game theory model to describe the author and the meeting and gave a negative answer to this idea. The research paper has been accepted by Economics and Computation (2022). Under this model, this article discusses the performance of different review and decision-making mechanisms in weighing meeting quality and review pressure, such as the following issues:

• How to determine the best Excellent acceptance threshold?
• Should we increase the number of reviewers on a paper?
• What are the benefits of improving the quality of review?
• Should the author also provide previous review comments for the paper?
• ......

Paper link: https://arxiv.org/pdf/2303.09020v1.pdf

1. Model Overview

This article models the process of authors submitting papers to academic conferences and reviewing them as a repeated game. The specific process is as follows:

First, each author has a paper ready for submission. In each round of submission, the author makes one of two decisions: submit the paper to a top conference or a sure bet (such as a less prestigious second-category conference). The results submitted to the IM conference and sure bet depend on the review mechanism and the quality of the paper:

• IM will have a certain probability of accepting the paper. Once accepted, the author will receive greater benefits. ;
• sure bet guarantees that the paper will be accepted, but the benefits will be small.

Among them, the decision to approve the review depends entirely on the reviewer's review opinions. For example, set an acceptance threshold and accept it if and only if the average review score is higher than the threshold. This paper, and the author's income decreases exponentially with the number of resubmissions.

The Dinghui promises a review/decision-making mechanism, and the author will make the best strategy for this mechanism; while the Dinghui needs to design a review and decision-making mechanism based on the best response strategy of the author. The optimal mechanism to balance meeting quality and review pressure.

2. Main conclusions

Using the above modeling method, this paper draws some conclusions Important conclusions, including:

1) The author’s optimal strategy

#In a simplified model (see the original text for more complex models), this article makes The following assumptions are made: authors know the true quality of their papers, conference decisions are memoryless (the decisions for each round of review only depend on the opinions of that round of reviewers), and authors have unlimited resubmission opportunities. In this case, the author has a threshold optimal strategy:

• #If the quality of the paper is higher than the threshold, the author will choose to submit the paper to the top review, and no matter how many rejections are experienced , the author will choose to resubmit until the manuscript is approved;
• If the quality of the paper is lower than the threshold, the author will immediately choose sure bet.

Normally the author's submission threshold Θ is lower than the conference's acceptance threshold τ, as shown in the figure below.

The above conclusion can be used to explain the resubmission paradox: why accepting more papers cannot essentially Reduce review pressure? This is because lowering the conference’s acceptance threshold τ will simultaneously lower the authors’ submission threshold Θ, thereby attracting more submissions of low-quality papers. As shown in the figure below, if the acceptance threshold is lowered, some papers (purple area) that were previously selected to be submitted to the second category conference are now selected to be submitted to the top conference.

2) Meeting quality and review pressure

The review/decision-making mechanism of the top meeting needs to weigh the quality of the meeting And review pressure, you can’t have both.

• Conference quality = the sum of the quality of all accepted papers
• Reviewing pressure = a paper from submission to final acceptance The expected value of the number of times the manuscript will be reviewed

Changing the acceptance threshold will change both the meeting quality and the review pressure (as shown below).

The picture shows the relationship between meeting quality (ordinate) and review pressure (abscissa) regarding the acceptance threshold Change curve, σ is the standard deviation of reviewer noise.

The following three situations can lead to a better trade-off between meeting quality and review pressure (less review pressure is required to achieve the same meeting quality):

• Better review quality——lower reviewer noise;
• lower review reputation————compared to Sure bet, the top will bring lower income;
• The author who is more short-sighted - the author's income will be greatly reduced in multiple rounds of re-investment.

3. Conclusion

This article aims to call on academic conferences to consider the incentives brought by different mechanisms to paper authors when improving their review and decision-making mechanisms. For more interesting conclusions, see the original text of the paper. For example, what factors mainly affect the acceptance rate of the paper? What is the optimal strategy for authors without accurate knowledge of the quality of their paper? What impact does requiring authors to provide previous review comments on a paper have on the conference?

Of course, the theoretical model of this article has many limitations at different levels: for example, this article does not consider the negative feedback effect of review pressure on review quality, and the impact of conference quality on author income. Positive feedback effect, and believe that the quality of the paper will not be improved during the rejection process, etc. The discussion and improvement of the conference peer review system will not stop here. It is particularly important to understand the conference review mechanism from a game perspective. Interested readers are welcome to view the original text of the paper or write to the author of the article to discuss more research details.

The above is the detailed content of Should Dinghui lower the acceptance threshold? Use game theory to explore optimal review and decision-making mechanisms. For more information, please follow other related articles on the PHP Chinese website!