| 1 | Jun 10, 2010 8:38 PM | Reviewers submit sample papers, key words and citations are extracted, and the system does smart matching. But the second phase of manually assigning some reviewers works well. |
| 2 | Jun 10, 2010 10:42 PM | It is important to be clear on whether first phase is only cursory or whether full reviews are expected. |
| 3 | Jun 10, 2010 10:46 PM | Combination of 'C' and 'D' where the papers are assigned based on the bids, but the area chair can overrule the matching. (AC's should be encouraged not to do it very often.) |
| 4 | Jun 10, 2010 10:47 PM | I don't like the two-phase system, because it means most accept/reject decisions are made without author feedback which at least places some limits on reviewing quality.
I believe we should start with a single phase model, where one reviewer is determined by area chair, one by reviewer bidding, and one by either or some other system. Diversity seems important here.
Then, instead of having a highly staged author feedback, we should simply have an anonymized email service between authors and reviewers, with an option for cutoff if abuse should occur. An area chair overseeing the process can bring in an additional reviewer as needed.
I also think we should differentiate the reviewers. One reviewer should be tasked with checking correctness (only), while others can assess significance. |
| 5 | Jun 11, 2010 12:59 PM | First review round with three reviewers assigned through bidding. Second round with manually assigned reviewers only for those papers that could not clearly be accepted or rejected. |
| 6 | Jun 11, 2010 1:01 PM | Mixing bidding and manual assignment but not in the sequence as this year. After the system has assigned papers to reviewers based on bidding, area chairs correct the assignment where they consider this necessary. |