Comment Summary

What model do you prefer for assigning reviewers to papers? (multiple selections possible)
#	Response Date	I have the following alternative proposal:
1	Jun 10, 2010 7:13 PM	This year's model, but in Phase 2 also use the reviewer bids to guide the assignments. (The software should be able to come up with a third reviewer for each borderline paper, just as it would have in a single-phase model.) I heard from ECML09 area chairs that it was very hard to do manual assignments because you don't know what expertise/interest the huge pool of PC reviewers has--- you tend to just assign them to a few colleagues that you know personally--- but that doesn't spread the load nicely.
2	Jun 10, 2010 8:38 PM	Reviewers submit sample papers, key words and citations are extracted, and the system does smart matching. But the second phase of manually assigning some reviewers works well.
3	Jun 10, 2010 8:39 PM	all are unimportant. before the reivews are given, reviewers should argue with themselves.
4	Jun 10, 2010 8:40 PM	If the NIPS topic model reviewing plays out well this year, you may want to assign first round reviews by that model and second round reviews manually.
5	Jun 10, 2010 10:42 PM	It is important to be clear on whether first phase is only cursory or whether full reviews are expected.
6	Jun 10, 2010 10:46 PM	Combination of 'C' and 'D' where the papers are assigned based on the bids, but the area chair can overrule the matching. (AC's should be encouraged not to do it very often.)
7	Jun 10, 2010 10:47 PM	I don't like the two-phase system, because it means most accept/reject decisions are made without author feedback which at least places some limits on reviewing quality. I believe we should start with a single phase model, where one reviewer is determined by area chair, one by reviewer bidding, and one by either or some other system. Diversity seems important here. Then, instead of having a highly staged author feedback, we should simply have an anonymized email service between authors and reviewers, with an option for cutoff if abuse should occur. An area chair overseeing the process can bring in an additional reviewer as needed. I also think we should differentiate the reviewers. One reviewer should be tasked with checking correctness (only), while others can assess significance.
8	Jun 10, 2010 10:55 PM	I think having only 2 reviewers is too few in the first phase. It also means that the authors do not get to respond to any additional reviews that come out after the phase 1 reviews. If there's going to be a response period, then I think it makes sense to try to get all the reviewers' comments before then. I'm indifferent whether the reviewers are assigned based on their bids or area chairs.
9	Jun 10, 2010 11:12 PM	A single-phase model where reviewers are assigned to papers based on their bids and the opinion of area chairs. So nothing is completely automatic but also not totally manual either. Also there should remain a chance of going to the second phase for some papers (say 20% of papers), but it should not happen for most papers. But there still should be a discussion phase.
10	Jun 10, 2010 11:36 PM	Again, why not use some machine learning?
11	Jun 10, 2010 11:55 PM	Single phase model, where reviewers are assigned to papers by area chairs, but if possible based on their bids.
12	Jun 11, 2010 1:08 AM	The new model can bias the reviewer in the second phase towards comments in the first phase. I prefer the traditional model where 3 reviewers provide comments, then the author responses, and finally the area chair makes final decision.
13	Jun 11, 2010 2:10 AM	The two-phase model was poorly implemented this year. On one paper, I specifically said something like "this paper is so bad it should not be given a second-round review", and the other reviewer's review was equally negative (both reviewers had very high confidence), yet the paper still got some second phase reviewing. If that's going to be the case, we might as well do a single phase review. Otherwise we need more aggressive first-round pruning, especially for such clear-cut cases (and I can't imagine a more clear-cut case than this particular paper!).
14	Jun 11, 2010 7:21 AM	NIPS is trying an automated system based
15	Jun 11, 2010 12:59 PM	First review round with three reviewers assigned through bidding. Second round with manually assigned reviewers only for those papers that could not clearly be accepted or rejected.
16	Jun 11, 2010 1:01 PM	Mixing bidding and manual assignment but not in the sequence as this year. After the system has assigned papers to reviewers based on bidding, area chairs correct the assignment where they consider this necessary.
17	Jun 11, 2010 1:20 PM	Unclear
18	Jun 11, 2010 1:26 PM	Get three reviews in phase 1 and only do extra reviews if there is a conflict.
19	Jun 11, 2010 1:33 PM	Allow (perhaps encourage?) Phase 1 authors to indicate (when applicable) they may not be approrpriate to review a particular paper after assignments have been made and replace them with a suitable substitute.
20	Jun 11, 2010 5:02 PM	Single-phase sounds simpler and saner, but it would be good to have more than 2 reviewers in Phase 1. I felt that one negative review our of two was overwhelming.
21	Jun 11, 2010 6:08 PM	Authors bid on reviewers
22	Jun 12, 2010 11:12 AM	NIPS idea looks nice. 1. propose say 15 papers to each reviewer based on the actual system 2. reviewer rank them 3. train a system to assign based on the ranking
23	Jun 13, 2010 12:00 PM	for a comment: I honestly do not know whether the 2-phase system makes such a difference - I have heard so many people say that reviews never really change after the first phase, even if there was a misunderstanding (and I think it is true). Another issue for good reviews is I think the number of papers per reviewer: honestly, I find 8 papers far too much and time-consuming to do an intense review, especially as a beginner. It would also be good if one could mix senior and non-senior reviewers, so that one paper does not only get one of each class.
24	Jun 13, 2010 10:07 PM	http://research.microsoft.com/apps/pubs/default.aspx?id=122784
25	Jun 14, 2010 3:54 PM	More automated (e.g., based on keyword matches) to eliminate friends choosing to review friends papers via bids. It's too easy to guess authorship from content in a subarea.
26	Jun 19, 2010 5:03 AM	Tauthor rebuttal should be transferred to the reviewers in Phase II.
27	Jun 25, 2010 6:39 AM	Assign papers based on match between paper's content (keywords) and the reviewer's expertise after eliminating conflicts, with some manual fine-tuning of assignments.