Netdev Program Review Guidelines

These guidelines have been been established to provide a fair and unbiased review process of program proposals submitted to Netdev Society.

Background

Netdev Society welcomes submissions for proposals for technical presentations at Netdev Society conferences. Proposals are to be judged for acceptance on their technical merit without bias or subjectivity. To this end, Netdev Society has instituted a blind peer review process. Proposals are reviewed by a program committee for inclusion in Netdev Society programs. The program committee is a set of volunteers that have demonstrated technical expertise and have made significant contributions to the Netdev community. It is important to note that the program committee performs it technical reviews independently of conference organizers that deal with logistics or work with sponsors. In particular, corporate sponsorship of a Netdev conference in no way guarantees that any proposals or presentations are accepted.

The procedures described here were motivated by the SIGCOMM peer review process as well as the consensus decision model of IETF. The procedures have been adapted and scaled appropriately to work for Netdev Society.

Roles

This section describes the roles in Netdev conference submissions and review process.

Submitters

Submitters are individuals that make proposals for inclusion in the Netdev conference program. A submission is done via the submission page. The content describes the proposal and identifies the submitters.

Submissions include the following information:

Name(s) of the submitter(s)
Title of the submission
Label (one of moonshot, nuts'n bolts, hands-on)
Submission type (one of talk , p resentation , workshop)
Estimate of length of time for presentation
Affiliations of submitters (needed for conflict of interest check)
Description of proposal

Please note that submitters must not communicate directly with any program committee members concerning their proposal. The exception is that a committee member is also submitter for the proposal or is otherwise intimately familiar with the work-- in such cases the program committee must be recused from reviewing the proposal. The program shepherd serves as the sole liaison between submitters and the program committee.

Program committee

The role of the program committee is to evaluate submitted proposals. For a each proposal, a subset of committee members are selected to review the proposal. Proposals are evaluated on their technical merits, suitability for the Netdev conference, appropriateness for the potential audience, and overall relevance of the work to the community. Ultimately, the committee makes a consensus decision whether to accept a proposal for a Netdev conference.

Program committee chair

The program committee chair is responsible for the overall flow and content of the conference. The chair is a member of the program committee and participates in proposal discussions. Generally, the chair will participate in discussions for all proposals (except in cases of potential conflict of interest) and will ensure that progress is made and that consensus is reached concerning acceptance for a Netdev conference. The chair may at their discretion reference other proposals they are aware of for the purposes of comparing similar proposals and promoting a good balance of presentations. Part of the chair’s responsibility is determining when consensus has been reached, however in normal technical discussions the chair should ‘take their hat off’ so that their opinion carries no more weight than other committee members.

Program shepherd

The role of the program shepherd(s) is the communications conduit between submitters and the program committee. The shepherd is not member of the program committee and does not participate in the decision process or technical discussions. The shepherd receives proposals from submitters and anonymizes them for sending to the program committee. If the program committee requires clarification from a submitter concerning a proposal, they will refer questions to the program shepherd who will then forward to the submitter. There may be more than one program shepherd for any conference to distribute load.

Program scheduler

The program scheduler handles the scheduling of presentations. They will work to appropriately schedule time slots for presentations to ensure a good ordering and to satisfy the requested presentation times. The scheduler may request adjustments to presentations to fit in allotted time.

Review panel

For each proposal a subset of program committee members is selected by the program shepherd to review the proposal and make a consensus decision about whether proposal is accepted. Minimally, a review panel will consist of four program committee members, however it may have more.

The shepherd will select the review panel for a proposal based on the following factors:

Potential conflicts of interest either declared or perceived
Areas of expertise of program committee members
Load balancing work amongst program committee members
Random selection

Submissions procedures

The Netdev submission and program committee procedures are as follows:

Create a new account
Check inbox for email verification link
Submit your abstract
The program shepherd receives the submission and verifies all the necessary information has been provided. An acknowledgment is sent the submitter and the proposal is registered in the proposal system.
The program shepherd anonymizes the title and abstract as necessary and sends out a conflict of interest query (COIQ) for the proposal to the general program committee mailing list(program-committee@netdevconf.info). The query includes the label, submission type, title, and description-- with the last two items being anonymized. Each member of the committee considers whether they have a potential conflict of interest if they were to review the proposal. For instance, if they recognize the work as being done by a colleague they are working closely with then that would be considered a conflict. Also, committee members may submit proposals in which case they would have an obvious conflict of interest for reviewing their own proposal.
Committee members reply directly to the shepherd as to whether they have a conflict of interest or would otherwise prefer to recuse themselves from reviewing a proposal. If a committee member has questions about potential conflict of interest then they should send those directly to the program shepherd. Note that there is no discussion about the proposal during the COIQ period.
Once all members have replied to the conflict of interest query (or seven days of elapsed) the program shepherd will select a group of members for the review panel of a proposal. The criteria for selection are described above in 'program shepherd role' section.
The shepherd initiates a review of the proposal. This is done by sending a Request for Review (RFR). The email includes the anonymized title, label, submission type and anonymized description. It is sent to the members of the review panel for the proposal.
The review panel discusses the proposal. They should do this on the same RFR thread initiated by the program shepherd. Only the members of the panel participate in the discussion and there should be no side discussions concerning the proposal with other members of the committee, the submitters, or anyone else outside of the review panel.
The review panel may have questions or require clarifications from a submitter. The panel creates a Shepherd Request (SR) email. The shepherd will follow up with the submitter and provide their anonymized responses to the panel.
The panel discusses the proposal until consensus is reached and an acceptance decision is made.
There are three possible decisions:
- Accepted - proposal accepted for inclusion in the conference.
- Accepted with contingency - proposal is accepted assuming that some contingencies are met. Examples of contingencies are: a request to the submitter to frame the subject in a certain way, eliminate overt references to commercial products, work with another submitter to ensure that potentially overlapping proposals are resolved.
- Not accepted - proposal is not accepted for inclusion
Once a decision is made, one member of the panel is chosen by the panel to write a Decision and Explanation (DAE) (this is not necessarily the program committee chair). This includes an explanation by the panel for its decision. If the proposal was accepted with contingencies then the panel will detail the contingencies. In the case a proposal is not accepted it is important to give useful feedback to the submitter for future proposals.
The panel's decision is sent to the shepherd. Unless there are procedural issues with the decision, the panel decision is binding. The decision is logged in the program system.
The shepherd informs the submitter of the decision and provides the explanation. If a proposal is accepted it will be posted on the web site as accepted.
If the decision is accepted with contingencies, the program shepherd will follow up with the submitter to make sure the contingencies are met. The program shepherd may also consult with the review panel concerning the contingencies.
Once a proposal has been accepted and any contingencies have been met, the program scheduler will schedule the presentation in the agenda. If any adjustments must be made to the length of presentation or other scheduling constraints considered then those will be negotiated between the program scheduler and the submitter. The program scheduler may also consult with the program committee chair or members of the program committee for optimizing the program agenda.