« Back to Summary
What model do you prefer for assigning reviewers to papers? (multiple selections possible)
#Response DateI have the following alternative proposal:
1Jun 10, 2010 7:13 PMThis year's model, but in Phase 2 also use the reviewer bids to guide the assignments. (The software should be able to come up with a third reviewer for each borderline paper, just as it would have in a single-phase model.) I heard from ECML09 area chairs that it was very hard to do manual assignments because you don't know what expertise/interest the huge pool of PC reviewers has--- you tend to just assign them to a few colleagues that you know personally--- but that doesn't spread the load nicely.
2Jun 10, 2010 8:39 PMall are unimportant. before the reivews are given, reviewers should argue with themselves.
3Jun 10, 2010 8:40 PMIf the NIPS topic model reviewing plays out well this year, you may want to assign first round reviews by that model and second round reviews manually.
4Jun 10, 2010 10:42 PMIt is important to be clear on whether first phase is only cursory or whether full reviews are expected.
5Jun 10, 2010 10:46 PMCombination of 'C' and 'D' where the papers are assigned based on the bids, but the area chair can overrule the matching. (AC's should be encouraged not to do it very often.)
6Jun 10, 2010 10:47 PMI don't like the two-phase system, because it means most accept/reject decisions are made without author feedback which at least places some limits on reviewing quality. I believe we should start with a single phase model, where one reviewer is determined by area chair, one by reviewer bidding, and one by either or some other system. Diversity seems important here. Then, instead of having a highly staged author feedback, we should simply have an anonymized email service between authors and reviewers, with an option for cutoff if abuse should occur. An area chair overseeing the process can bring in an additional reviewer as needed. I also think we should differentiate the reviewers. One reviewer should be tasked with checking correctness (only), while others can assess significance.
7Jun 10, 2010 11:55 PMSingle phase model, where reviewers are assigned to papers by area chairs, but if possible based on their bids.
8Jun 11, 2010 1:08 AMThe new model can bias the reviewer in the second phase towards comments in the first phase. I prefer the traditional model where 3 reviewers provide comments, then the author responses, and finally the area chair makes final decision.
9Jun 11, 2010 1:01 PMMixing bidding and manual assignment but not in the sequence as this year. After the system has assigned papers to reviewers based on bidding, area chairs correct the assignment where they consider this necessary.
10Jun 11, 2010 1:26 PMGet three reviews in phase 1 and only do extra reviews if there is a conflict.
11Jun 11, 2010 5:02 PMSingle-phase sounds simpler and saner, but it would be good to have more than 2 reviewers in Phase 1. I felt that one negative review our of two was overwhelming.
12Jun 11, 2010 6:08 PMAuthors bid on reviewers
13Jun 12, 2010 11:12 AMNIPS idea looks nice. 1. propose say 15 papers to each reviewer based on the actual system 2. reviewer rank them 3. train a system to assign based on the ranking
14Jun 13, 2010 12:00 PMfor a comment: I honestly do not know whether the 2-phase system makes such a difference - I have heard so many people say that reviews never really change after the first phase, even if there was a misunderstanding (and I think it is true). Another issue for good reviews is I think the number of papers per reviewer: honestly, I find 8 papers far too much and time-consuming to do an intense review, especially as a beginner. It would also be good if one could mix senior and non-senior reviewers, so that one paper does not only get one of each class.
15Jun 14, 2010 3:54 PMMore automated (e.g., based on keyword matches) to eliminate friends choosing to review friends papers via bids. It's too easy to guess authorship from content in a subarea.
16Jun 19, 2010 5:03 AMTauthor rebuttal should be transferred to the reviewers in Phase II.
17Jun 25, 2010 6:39 AMAssign papers based on match between paper's content (keywords) and the reviewer's expertise after eliminating conflicts, with some manual fine-tuning of assignments.