We thought we'd share these experiences with you, in case you find yourself in need of similar features. Briefly, we settled on Zoom, but Zoom didn't solve all of our problems. At this point we are mainly interested in reporting our experiences while they are still fresh in our memory, in the hope you will find it useful: it would take more time and experimentation to turn this document into a concrete set of recommendations.
Running this conference is a relative complex operation for many reasons including these. ATC received hundreds of submissions and the full set of reviewers (in three tiers) is almost 120. And there's two co-chairs plus two submission chairs, all with the necessary administrative privileges in the online paper reviewing system (HotCRP.com -- yes, often pronounced Hot Crap :-).
Webex does have a decent waiting room feature: we could manually move attendees to the waitroom and we verified that they cannot hear/see anything, and cannot get back on their own.
Webex has a very nice registration feature: you invite N people with specific emails and names to a Webex meeting, they require you to register with the email they were invited with, and they cannot change their name.
Bluejeans supports a "breakout room": we can move people to the breakout room. Alas, people in the breakout room can then rejoin the main meeting on their own--clearly undesired. (I guess it's like a conflicted PC meeting member, who is outside the main room, barging right back in.)
We didn't test Bluejeans's registration feature, as the breakout room mis-feature was a showstopper for us.
Zoom's waitroom worked very well. Participants with conflicts could be kicked out of the meeting and go into the wait room, where they can't hear/see anything. We could then re-admit all them with a single click on the admit-all button; then we kick out of the meeting the next set of CoI. The key here is that *all* co/hosts were able to manage these conflicts and the waitroom, allowing us to better parallelize (and double check) this complex task.
Zoom's registration feature is not as good as Webex's. We have to send the zoom url to all our PC members. They each have to register with a valid email and put their names. They then get an email with a personal link to join the meeting (so, thankfully not a shared URL that can be easily zoombombed). But, they can put in any valid email and any first/last name. In theory, someone could create a new dummy email and masquerade as another PC member, if they got their hand on the invitation URL. So we're going to have to ask them to use their proper names and emails that are registered in HotCRP: when the meeting starts, all PC members will be in the waitroom by default, and we'll have to verify one by one whom we are admitting into the meeting---else we can chat privately with them in zoom "who are you?" Once we admit everyone, we'll turn off the feature that "participants can rename themselves".
This registration becomes even more important for people who will dial in by phone to the meeting: they still have to register with a per-participant link, then they get an email with instructions how to connect to the meeting with a personal phone code that identifies them. When users who dial in connect, they are shown as "Phone User n": we have to identify them by voice and then rename them in the Zoom's Participants list so everyone knows who they are.
We've also heard about at least one PC meeting via Microsoft Teams, which reportedly worked out well. But we were not sure we have access to test it, unless we have Office 360 that includes it. Since we were reasonably pleased with the Zoom setup, we didn't investigate MS Teams (anyone with detailed experiences, please share).
I, Erez, had a chance recently to join three different back-to back meetings with about 6-8 people each: Microsoft Teams, Zoom, and Google Meet. Overall Zoom worked much better: I don't think Teams or Meet would have [sic] met our vPC needs.
Microsoft Teams does seem to have a wait room feature, as I was waiting to be admitted, but I'm not sure how well it would work for running a vPC. Video and audio quality was lower for some participants, so I worry it would not have scaled to our PC size; but it could be their own Internet connection. I could only see 4 people's videos at a time, even though there were others with video on: that limits the ability to feel "inclusive" and see more people. And while I was at the meeting, I also looked at all the buttons and menu options: it seems that Teams had far fewer features.
Google Meet also had very few features compared to Zoom, even fewer that Microsoft Meet. The worst part was that audio and video quality under Google Teams was considerably poorer for everyone participating; even when turning off everyone's video and just steaming audio, quality was still fairly choppy.
When streaming media for hours, some people's computers overheated and shut down after a few hours. Be sure you have a powerful enough computer for long-running CPU-hog processes like video+audio streaming.
We used Slack as a "side" channel for private communications among the meeting runners. We could have used Zoom's chat features but it was too risky -- participants could inadvertently broadcast something publicly unintentionally. So we allowed participants to chat only with the host(s) in zoom. It was useful as people had to tell us about last minute schedule changes or other requests. The Zoom messaging feature was not very convenient, however, when needing to send the same message to few participants (but not all, so as not to violate conflicts), for instance that their paper will need to be reshuffled in the schedule. Also, Zoom let participants chat with one of the co/hosts, but not all of them as a group. Lastly, there was no way to clear the chat history between paper discussions, so as to avoid leaking information to other participants once they rejoin.
While zoom permitted us to manage conflicts as described above, it took time to do so: we had to look up the conflicts in hotcrp, then scroll or search for the right participant in the participant list, and one by one move them to the waiting room. There is no feature for participants to take themselves into the waiting room, the way they would during an in-person PC meeting. Zoom, perhaps under network stress, had a delay of 2-3 seconds between when you kicked someone off the meeting and until they actually showed up in the waiting room. So it took 1-2 minutes per paper just to manage those conflicts, precious time when you are under a tight schedule. Conversely, in a physical PC meeting, you quickly call the names of all conflicted members, and they all get up at once and leave the room "in parallel."
Zoom shows at most 25 participants' videos at once. And not all our participants used their video. (One insisted on calling in from an "anonymous" phone number due to reports of Zoom privacy concerns.) So it was harder for PC members to know when they can jump in and speak. We tried to manage the order as best we could, calling on people in turn; and we also used the "raise hand" feature a bit. But it still took longer than with an in-person meeting. Plus there were natural delays in people's audio/video stream, few people with poor connections, etc. All this added another 1-2 minutes of time when discussing each paper.
When a PC is held in person, people come from all over the world and would be at the start of the PC at the designated time. But with a virtual PC spanning 12-15 time zones, it was impossible to expect people to be at the meeting at ridiculous early/late hours. So our meeting was scheduled for the middle of the day or so. We sent a doodle survey to see what times people can attend. And we tried our best to group papers based on people's time constraints -- not easy. Worse, because of COVID19, people had day-job duties they couldn't ignore, childcare duties, last minute schedule changes, and more. So we had to adapt to people's changing schedules dynamically. This added more "context switching" time between papers.
Few other aspects made the process more challenging. First, it was more difficult to control inadvertent leakage of information about paper reviewers -- we had cases where either one of us or reviewers themselves asked if we can do paper #X before they leave, or when we waited to discuss a paper because of a missing reviewer, but now that information is visible to others -- they see who just joined the meeting. Likely some of this exists in an in-person PC meeting, but probably less so. Second, managing the discussions to wrap up in a fixed amount of time was more difficult, given the lack of other options. PC voting as an option really doesn't work in an online format. We rarely had the full PC, and with people coming and going and videos switched off, it was difficult to tell who's around, whether they would listen in a brief summary of the discussion before voting, etc. As a result, for cases when the PC discussion is "deadlocked" and it is obvious that a reviewers' vote won't resolve it (e.g., an even number of reviewers split 50/50), asking the PC to vote cannot resolve the paper's status. In addition, it's harder to ask the PC members to take the conversation offline and report back -- something that's commonly done during in-person PC meetings -- because of the above-mentioned issue with time zones and daytime duties. Taking a conversation "offline" meant really pushing papers to be decided at some undetermined later point, likely after the actual PC meeting. These two issues made it harder to cut discussions short, which again added to the meeting time.
We already expected that our virtual PC wasn't going to be as effective as an in-person one would have been. And so for weeks leading to the PC meeting, we pushed our PC hard to try and reach a decision on as many papers as possible. That certainly helped a lot (and we have even heard of some PC Chairs who canceled their online PC meeting so they don't have to deal with the complexities of running it virtually). Still, all these complications caused our PC meeting, originally scheduled for five hours, to take seven hours. And we still had a few of the discussed papers to finalize offline after the meeting.
Finally, a word about security and privacy. Since Zoom saw its user base grow 20-fold in just a few months, it has attracted a lot of media attention: some found and reported serious security and privacy concerns. (This is not to suggest that Zoom's competitors' security and privacy practices are perfect and their software bug free.) As a result, a few high profile communities (e.g., school districts) banned or abandoned Zoom altogether. To their credit, Zoom has apologized publicly, has begun to address these concerns, and already released several security fixes and new features, promising more. Still, some of our PC members, understandably, preferred not to run the Zoom client or accept their privacy policy (as there are reports of numerous Zoom users' credentials sold on the dark web); these users instead called in via phone.
---
Sincerely,
Erez Zadok and Ada Gavrilovska
USENIX ATC 2020 Co-Chairs