**4. Discussion**

#### *4.1. Opportunities in Virtual Stakeholder Engagement*

Overall, hosting the stakeholder meetings virtually using Zoom and GoogleDrive was successful. After participating in the workshop, stakeholders had the opportunity to complete an evaluation form to let us know about their experience, responses included:

• "I appreciate how difficult this was over Zoom, how difficult surveys are in general and how difficult trying to ge<sup>t</sup> consensus on the topic of microplastic can be. I found the discussion interesting and informative for our own work communicating about this problem. I look forward to seeing the results!"

#### *Societies* **2020**, *10*, 98


Conducting in-person social science research can be costly in terms of funding and time. Expenses include room reservations (multiple if breakout groups are necessary), mileage reimbursement for participants, hotel reservations for experts, video-recording equipment, food, and supplies. Given the limited funding in social sciences, these factors can be a major deterrent from being able to conduct stakeholder workshops. Additionally, Shelley-Egen emphasizes that there is a suite of technological alternatives already in existence [11]. However, most of those alternatives were created in conjunction with face-to-face meetings [11]. Thus, a challenge for this research was to utilize platforms that would eliminate the need for in-person interactions.

We chose technology platforms that were not only familiar to participants but also at a low cost to us (GoogleDrive services were free and the upgraded Zoom account was already provided to professors from the University). There were no costs associated with on-boarding participants to these platforms since Zoom is free and GoogleDrive allows the host to send materials for review and manipulation by non-Gmail users. There were also no expenses associated with travel by hosts, experts, or participants. Oftentimes, the funding constrains the number of people who can participate given the associated costs per person. With this virtual workshop, we were able to invite as many stakeholders as appropriate. Additionally, over the three days, only 10 percent of participants recruited did not show up, whereas 30 percent of participants recruited for previous work did not show up [16]. Altogether, using virtual platforms to conduct social science research removes the constraints detailed above and may provide an opportunity for the direction of future social science research.

We acknowledge that the success of this methodology is constrained by internet connectivity and quality and users comfort with technology and remote meetings. Belzunegui-Eraso and Erro-Garces emphasize the important role technology plays in the successful development of telework [27]. According to Belzunegui-Eraso and Erro-Garces, the ideal teleworker is a "Millenial woman holding a higher education degree, with 4–10 years of professional experience, and working from home two days a week in the managemen<sup>t</sup> and administration field" whereas the least ideal person is a "man of the baby boomers' generation, holding a university degree, with 20 years or more of professional experience, and who started working remotely only during quarantine" [27]. However, conducting human-centric research does not always allow for the ideal teleworker. Thus, this [meaning, technology literacy] is an important indicator of successful research during these COVID-19 times; future work should keep in mind socioeconomic status and geographic location of participants to assure that data is irrespective of technological challenges. For our research, we assumed that the participants were technologically literate, given their professions and backgrounds, but we also provided an opportunity prior to the workshop to confirm and adjust based on their technological literacy as needed.

Participants were required to view presentations ahead of their participation date. The presentations were from experts in the field of microplastics and oysters as well as one from the host explaining the general agenda of the workshop as well as the indicators that were to be evaluated by participants day-of. Typically, these presentations would have been done at the beginning of the workshops; however, in the initial discussions with participants about changing the format of the workshops from in-person to online, participants expressed that their time availability had become truncated, in essence due to other work obligations and childcare. Taking that into account, we shortened the time of the workshop from 5 h to 3 h and moved certain agenda items (e.g., expert presentations) prior to the workshop to be sure that participants had enough time for group deliberation. As a result of this, we may have encouraged more folks to participate given that they did not have to travel to Boston in order to participate (as was the plan during initial outreach). As of March 2020, we had 20 stakeholders agree to participate in the in-person workshop (30% from the Academic sector, 30% from Government, 25% from NGO, and 15% from Industry). At the completion of the virtual workshops, 30 stakeholders had participated (21% from the Academic sector, 41% from government, 24% from NGO, and 14% from Industry). This increase in stakeholders could be attributed to increased outreach and the virtual nature of the workshops. However, given childcare constraints, as discussed in the previous section, some may have still been discouraged to participate.

The change from in-person to virtual was communicated via email through the description of the three participation phases:


The expert presentations were uploaded to a shared GoogleFolder and shared with participants 5 days prior to their participation date. One participant said "I liked participating, it was very interesting and the format was fine, nice job! I am happy to have been part of this. I liked being able to listen to the information ahead of time and to hear a range of comments from such a group of stakeholders ... " When asked if participants were able to successfully view the presentations, 83 percent of participants indicated 'yes,' 10 percent indicated 'partially' and 7 percent indicated 'no' (Figure 4A). All participants found the instructional material provided was helpful either in totality (67 percent) or partially (33 percent) (Figure 4B). Additionally, participants were exposed to the key points of each experts' pre-workshop presentation at the start of Phase II; participants were o ffered the opportunity to ask questions and/or clarify points made by the experts at this time. For future work, one participant suggested that the videos be placed on a secure Youtube Channel rather than GoogleDrive. Furthermore, click rates could be monitored to assure that the videos were actually viewed ahead of the workshops and an additional question could be added to the post-workshop evaluation form asking participants if they viewed the material undistracted or while doing other work.

When these workshops are video-recorded in-person, it is evident to participants that a camera is recording their conversations and movements. This may become a hindrance to 'camera shy' individuals. With this new virtual format, participants were read a consent form pertaining to the use of video-recording at the start of the workshop. The camera was not as apparent throughout the virtual workshop (as it would have been in-person) which may have allowed those individuals to participate more comfortably.

We also decided to keep the daily groupings smaller in number compared to doing it in-person. Acknowledging that free-flowing conversation is not as easy in a virtual workshop, we wanted to make sure that all voices could be heard. Aided by a facilitator, participants were called on to help move the conversation forward when needed. One participant mentioned that they "appreciated the moderator calling on people to speak up. Often one person dominates the conversations, especially during remote meetings ... Getting more people to speak can only improve the discussion ... ". Even with the virtual format, however, participants were generally satisfied with the outcome of the group deliberation (Figure 4C).

**Figure 4.** Responses from the evaluation form that participants were asked to complete post-workshop.

#### *4.2. Challenges in Virtual Stakeholder Engagement*

At the crux of stakeholder engagement, and specifically the DMCE methodology, is the ability to interact with fellow participants. This interaction is important for both the researchers and the participants. For the researchers, the qualitative data (e.g., the discussions between participants) are just as, if not more important, than the quantitative data (e.g., the ranking values given to the scenarios). We wanted to understand the reasoning behind choices (why did participants preferences change during the group deliberation when compared to pre-deliberation preferences?). For the participants, being able to learn from other participants during the group deliberation is essential to the learning process that takes place while implementing the DMCE methodology. While using Zoom was a decent substitute for in-person relations (one participant said "We are all becoming more comfortable and competent with Zoom. It will be a most valuable tool in the future."), not all participants were able to use Zoom as a result of work-related mandates (one participant, who had initially agreed to participate, had to cancel because of the use of Zoom). Since participant and data security is integral to working with human subjects, using platforms that provide that is essential. Based on this experience, we can identify how we need to improve technological tools that we used in this process.

Lacking these in-person relations was one of the major challenges of this research. Previous work done using this methodology emphasized the need for the in-person relations [16]. Typically, participants would have been provided with a printout of the scenario cards in addition to the scale bar so that individuals could move the cards based on their preferences in real-time and others could evaluate and sugges<sup>t</sup> changes, thus prompting conversation among participants. In hopes of providing some of that tactile learning, we provided images of the cards on a GoogleSlide (that all participants had access to) so that they could move the cards in real-time. What we found, however, is that most participants did not utilize that function during the group deliberation section of the workshop and thus we required a graduate student to help move the cards based on the discussion.

One of the unique aspects of this methodology is being able to evaluate participant preferences before, during, and after knowledge consumption in order to see if preferences change or stagnate with the gaining of more knowledge. Typically, all of the information would have been provided day-of so that we were able to record initial preferences in real-time, rather than allow the participants to contemplate the information for days prior. However, as mentioned above, we had to adapt to current times by shortening the day-of workshop and providing materials ahead of time. Given that materials

were provided 5 days in advance, participants technically had the opportunity to reflect on the content, specifically the indicators, prior to completing the initial, individual survey day-of. For future work, participants could be asked to complete the initial, individual survey when they received the materials in advance.

Providing opportunities for participants to gain knowledge throughout the workshop is integral to successfully implementing this methodology. Knowledge building can happen during structured expert presentations, conversations with other participants, as well as candid conversations with experts throughout the day. For the latter, participants are told that experts are available throughout the workshop to clarify any points/questions. While experts were present during each virtual workshop for its entirety, participants oftentimes forgot they were there since they (the experts) are not meant to interject unless called upon, which may have led to the discrepancies in how influential the scientists ('experts') were on the outcome of the group deliberation (Figure 4D).
