response_reviewers.tex

\documentclass{article}
\usepackage[utf8]{inputenc}
%\usepackage{standalone}
% \usepackage[breaklinks=true,colorlinks=true,linkcolor=blue,citecolor=black,urlcolor=blue]{hyperref} % url
\usepackage[top=0.85in,left=1in,footskip=0.75in]{geometry}
\usepackage[dvipsnames]{xcolor} % colors
% \definecolor{blue-pigment}{rgb}{0.2, 0.2, 0.6}
\usepackage[colorinlistoftodos,textwidth=4.3cm]{todonotes} % for comments
\usepackage[normalem]{ulem} % for sout (strikeout)
\usepackage{url}
\usepackage{soul}
\usepackage[colorinlistoftodos]{todonotes} % for comments

\usepackage{lineno}
%\usepackage{xr-hyper} 
\usepackage{xr} 
%\usepackage{hyperref} 
\newenvironment{Reply}{\noindent\color{BlueViolet}\textbf{Reponse.}}{\vspace{1em}}

%\usepackage{xr} % for getting lines with modifs from the other doc
%\externaldocument[ms-]{plos_latex_template}
% In your preamble

\makeatletter
\newcommand*{\addFileDependency}[1]{% argument=file name and extension
  \typeout{(#1)}
  \@addtofilelist{#1}
  \IfFileExists{#1}{}{\typeout{No file #1.}}
}
\makeatother

\newcommand*{\myexternaldocument}[1]{%
    \externaldocument{#1}%
    \addFileDependency{#1.tex}%
    \addFileDependency{#1.aux}%
}
\myexternaldocument{plos_latex_template}
%%%% defining colors for coauthors %%%%

\setlength{\parskip}{0.3cm plus 4mm minus 3mm} % space between paragraphs

\title{Ten simple rules to host an inclusive conference\\ \vspace{0.5cm}
\textbf{Response to reviewers}}
\author{}
\date{}


\begin{document}

\maketitle

\section*{Note from the authors}

Dear Dr. Markel, 

Thank you for considering our article for publication. We would like to thank the reviewers for their detailed feedback and thoughtful comments and suggestions.

Please find the detailed responses to all the reviewers' comments below. We added line numbers and quotes and highlighted our changes in the manuscript. 
Comments from the Reviewers led to editing and paraphrasing of several parts of the manuscript, including
i) direct mentions of the involvement of the community when defining diversity and inclusion priorities (in Rule 1),
ii) ways to ensure accountability during the organizing process and after the conference (e.g.\ in Rule 2 with a Diversity, Equity, and Inclusion (DEI) team, Rule 8 with guidelines like the diversity statement and the code of conduct, and Rule 10 with post-conference continuity),
iii) mention of women in science in Rule 2,
iv) biases due to video abstracts, 
v) the connection between the code of conduct and the diversity statement, and
v) the process we followed to put together this list of simple rules---now described in detail below and summarized in the current manuscript version. We also included the sets of rules gathered by the team as Supplementary Information. 

While writing and revising the manuscript, we tried to minimize references to the useR! experience. Though the conference was the primary source of inspiration and lessons learned and implemented in this manuscript, we wrote the rules in a way that could be applied to conferences in all regions and areas of knowledge. 

To be consistent with our own rule of making room for linguistic diversity and to ensure accessibility across the scientific community, we translated the extended version of the abstract and list of rules into ten other languages. We now mention them at the end of the English abstract. The translated abstracts are also included as Supplementary Information. To ensure broader reach, we would like to make a special request for them to appear on the main page of the manuscript online.

We sincerely hope that you will now find our revised manuscript suitable for publication. Please let us know if you require any additional information. 

Sincerely,
on behalf of the authors,

Roc\'io Joo and Janani Ravi


\section*{Editorial Summary} 

Dear Dr. Joo,

Thank you very much for submitting your manuscript `Ten simple rules to host an inclusive conference' for consideration at PLoS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

\begin{Reply}

    Thank you very much for this good news, and special gratitude to the members of the editorial board and the independent reviewers for the positive feedback.
\end{Reply}

% When you are ready to resubmit, please upload the following:

% [1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

% [2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

% Important additional instructions are given below your reviewer comments.

%  Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.


\section*{Reviewer \#1:}
I greatly enjoyed reading the paper `Ten simple rules to host an inclusive conference', and think it provides an easy to follow set of rules that organizers can use to ensure an inclusive conference. I found many tips for inclusiveness in the article to be especially helpful, such as multilingual accommodations and ensuring screen-reader accessibility.

I just have a few aspects I wasn't 100\% clear on.

(1) In Rule 4, the manuscript states `Define measurable indicators for these goals e.g.\ a gender distribution of your speakers that is representative of the general population, the participation of people from diverse races and ethnicities among the organizers, speakers, and attendees, or participation from key geographic regions', but how do organizers choose the correct demographics to monitor and the expected background rates? Should it match global, the local, or field-specific rates? Or should it be remediative, where it specifically selects for communities historically ignored in the field beyond the regional population rate?

\begin{Reply}

    % RJ: This is actually rule 1.
   The reviewer asked very important questions, for which there are no straightforward answers or formulae. 
   The `correct' demographic will vary for every conference, field, and local/regional contexts. 
   We believe that while remediation measures should be implemented in all conferences, every meeting committee or organizing team will have to ask themselves these questions and formulate their own inclusive plan.
   We now explicitly mention the need for this reflection and suggest involving the community as well as using available information on diversity in the field (lines \ref{indicators-1} to \ref{indicators-2}):
   
   \textit{
    Lack of diversity will vary across areas of knowledge and depend on the geographic context of the conference.
    Assess how it translates to the conference's particular field and scientific or professional community.
    Some information can come from Diversity, Equity, and Inclusion (DEI) studies in the field or at a larger scale (e.g. STEM---science, technology, engineering, and mathematics) if they exist. 
    Most importantly, involve the community---the subset of the general public that would ideally attend the conference---to help make this assessment and set the vision and goals of diversity and inclusion for the conference.
    }
   
\end{Reply}

(2) Adding on to the first point, how are the indicators measured? Should the conference organizers quantify the demographics of the attendees/participants? 

\begin{Reply}

The evaluation of goals and outcomes is part of Rule 10, and has now been addressed in the manuscript in lines \ref{info} to \ref{info2}:
\textit{
(...) You will likely be able to evaluate the outcomes associated with these goals directly from the information collected from the organizing team, speakers, and participants (e.g.\ city/state/country of origin or preferred language).
You may have set goals concerning how welcomed and included participants felt during the conference, and need to assess if the implemented practices translated into true inclusion. 
Surveys, focus groups, and informal conversations during and after the conference are good ways to collect this information and great opportunities to receive feedback from the attendees.
}
We also added in Rule 1 an explicit link to Rule 10 in line \ref{rule10} to let the reader know that the evaluation results will be used as part of the conference wrap-up and continuity. 

\end{Reply}



If so, should these details be published to the participants after the conference? 

\begin{Reply}

Yes, the organizing team should share the final assessment of the success of their inclusive actions with everyone who attended, helped create these goals, or could be part of future conference committees. 
We added more details about this in Rule 10, lines \ref{data-share} to \ref{data-share2}:
\textit{The feedback received from the participants and the whole assessment of the conference should be documented and shared with those who helped define the inclusion goals, the attendees, and future organizers. 
Ensure that the participants' privacy and anonymity are respected; ask for consent and consider binning/grouping the data into categories when there is a low number of respondents.}
\end{Reply}

How do organizers hold themselves accountable? And to whom are they held accountable to; the organizing committee themselves or the greater community?

\begin{Reply}

Conferences are organized for the community; thus, the organizers should hold themselves accountable to the whole community and not just the organizing committee themselves.
As we mentioned in the manuscript, this should be done along the way (line \ref{rule10}), during the organization process, by being transparent and communicating inclusion efforts and limitations (lines \ref{accountability-1} to \ref{accountability-2}):
\textit{Publish the diversity statement, code of conduct, accessibility guidelines, and options for financial support (see \textbf{Rule 9}).
These guidelines will not only communicate the conference values and practices but will also act as a built-in mechanism to hold the organizers accountable.
Be transparent with potential attendees, communicate the limitations of the conference to let them know what to expect, and outline ways in which the organizing team will try to mitigate these issues. 
For instance, inform participants if the conference platform is not completely screen-reader friendly and let people know if you are offering help to navigate it, or highlight on the conference website if captions will be available for some talks but not all.
Provide a point of contact to help clarify any questions regarding accessibility, financial support, and other issues affecting participation.}

As stated in Rule 10 (lines \ref{account-3} to \ref{data-share2}), there should be a final assessment of the achievement of the diversity and inclusion goals at the end of the conference. Many tools can be used for that (e.g.\ demographic information, surveys, focus groups, information from informal conversations). 
Publishing these final assessments is an important way to hold the organizers accountable both to the greater community and the committee itself.
\end{Reply}

(3) In Rule 4, it states `Check the final lists of speakers and do not hesitate to curate them to correct any imbalances you observe;' Does this pertain to invited speakers only, or speakers selected from submitted abstracts? If this pertains to scored abstracts, how does one ensure that this is done in a transparent and reproducible way? Or maybe I'm not fully grasping what curate means in this context?

\begin{Reply}

This specific section refers to the final selection of submitted proposals. Counteracting bias in invited roles was mentioned earlier in the text in lines \ref{invited} to \ref{invited-2}.
We agree with the reviewer that `curation' was an ambiguous term.
We rephrased the section as follows: \textit{
Even well-meaning and carefully crafted processes can be prone to reproduce the biases present in academia and society. It is, therefore, essential to examine the final list of selected abstracts to check if there is a lack of diversity. If proposals with similar quality were still judged differently because of unconscious bias, consider giving preference to the work from people in minoritized groups.
This manual examination at the final stage is unlikely to be reproducible. Nonetheless, the whole process should be as transparent as possible: evaluation criteria should be clearly stated and shared with the reviewers and authors, preferably during the call for abstracts.} (lines \ref{criteria-1}-\ref{criteria-2}).

\end{Reply}

(4) In Rule 5, it states `The physical venue would have to include multiple ways to connect with online folks such as cameras and microphones to allow them to follow who is speaking at the in-person space, and tablets to help in-person attendees interact with remote participants.' How do the tablets help in-person attendees interact with remote participants? Especially participants that may be in a different time-zone. Or are the tablets used to help the online activities mentioned in the next paragraph?

\begin{Reply}

Although one can assume that an ever growing proportion of people have their own handheld device to interact virtually with remote participants, we cannot assume that everyone does, or that the models are compatible with the software platforms used in the conference. 
If the conference organizers provide such devices, Q\&A sessions and networking activities can be held virtually and allow for more integration between in-person and online participants. We edited the manuscript for clarity: \textit{The physical venue would have to include multiple ways to connect with online folks [...], and tablets to help in-person attendees without suitable personal devices to interact with remote participants during Q\&A or networking sessions} (line \ref{tablets}).

\end{Reply}

(5) In Rule 7 it states `Allow for abstract submission in both English and the language the person feels more comfortable with, and consider the possibility of assigning a reviewer who is fluent in that language.' It is not trivial to submit an abstract in two languages, and if there is no chance that the non-English abstract won't be reviewed I feel like this would be wasting their time. Is it possible to denote the accepted languages ahead of time? Or for the submitter to put in a language request before the abstract deadline?

\begin{Reply}

    Thank you for this suggestion. 
    The organizing team should decide whether to ask for two languages or just one, depending on their capabilities. 
    Indeed, it is not trivial to submit abstracts in two languages. However, from personal experience, we know that many people find English so cumbersome that, if given the choice, they would opt for adding a version in their native language to increase the chances of communicating their ideas accurately. 
    The text now states: 
    \textit{Allow for abstract submission in both English and the applicant's preferred native language, and whenever possible, assign a reviewer who is fluent in that language. 
    Alternatively, the organizing team could define a set of accepted languages for abstract submission, and allow applicants to choose from this list. 
    The goal should be to judge the abstracts primarily by the quality or relevance of the work instead of English proficiency} (lines \ref{abstractlanguage1} to \ref{abstractlanguage2}).
    
    We also added a mention to accepting multilingual submissions when counteracting self-selection during the call for abstracts in Rule 4, linking to Rule 7 (lines \ref{multilingual-submission} to \ref{multilingual-submission2}).
\end{Reply}

Overall, I think these rules are great. My biggest concern is on the concept of organizing committee accountability. They are defining what the diversity goals are and how they are measured, but it isn't clear to me what input the community has on the conference. I feel like the greater community should be able to have input into the organizing process as well as the ability to see if their concerns are being addressed.


\begin{Reply}

    Thank you for the thorough and positive review. We hope to have addressed all of your concerns and particularly accountability. We agree with involving the community in every part of the process in an active and participative manner, using different channels of open communication and integrating members of the community into the organizing team. 
\end{Reply}


\section*{Reviewer \#2:}
L78-L80 As a female scientist I would ask myself "where are we" (because we still have parts of the World where female scientist are rare on leading positions), but I consider science apolitical in every segment, and I certainly suggest that it concentrate here on the scientific youth, the segment of research in industry, etc., and not on the above.
The rest of the paper it should certainly be praised because it enters into the essence of the exchange of knowledge today.

\begin{Reply}

    We thank the reviewer for this comment. The list given was meant to be an example of existing thematic groups and communities, aiming to go beyond gender representation.
    We have now added women as an example in line \ref{wis}:
    \textit{Open the schedule for events organized by thematic groups and communities (e.g.\ task force meetings, LGBTQIA+-friendly spaces, Black in STEM, women in science), and sessions aimed to welcome first-timers to the conference.}
\end{Reply}

\section*{Reviewer \#3:} 
This is a timely, well-written commentary on organising an inclusive conference and should act as a reference compendium for researchers, practitioners as well as conference organisers.

\begin{Reply}

    We thank the reviewer for the positive comment and the detailed suggestions.
\end{Reply}

I have some comments for authors to consider, and possibly expand upon.

1. As much as I enjoyed reading the ten rules, I missed finding mention of how this is to be implemented and governed. Therefore, it is poignant that authors should brief discuss having an inclusion oversight or governance committee that tracks these metrics and their implementation.

\begin{Reply}

    A Diversity, Equity, and Inclusion (DEI) committee would be best suited to ensure that the inclusion vision and goals are achieved. 
    Establishing a DEI team is now suggested in Rule 3 in the manuscript (lines \ref{subteam-1} to \ref{subteam-2}):
    \textit{ 
    Having a dedicated DEI team may be an excellent way to give the topic the attention and energy it requires.
    The core team should support the DEI initiatives and advocate for inclusion across all conference areas.} 
    Thank you for pointing this out. 
    On the other hand, the existence of a DEI team is neither necessary nor sufficient, and we believe the most important thing is the commitment of the 
    whole organizing team towards inclusion, and the willingness to hold themselves accountable, with concrete actions mentioned in Rule 8 (during the conference, lines \ref{accountability-1} to \ref{accountability-2}) and Rule 10 (after the conference, lines \ref{account-3} to \ref{info2}) .
    
\end{Reply}

2. Secondly, I also missed detailed methodology on how these ten rules were developed or consensus were reached at? Were stakeholders from diverse backgrounds consulted? Was the consultation process iterative? Were the recommendations thus made by those from diverse backgrounds incorporated? How were the final draft reached at - through consensus or voting or a process to that effect? This is important as for such a set of indications to be indicative, it is important that opinions from all backgrounds are taken into account and co-developed.

\begin{Reply}

    The rules were first drafted by the first author, based on the collective experience of organizing useR! 2021 and discussions with other organizers; then, the core group and the members of the diversity, accessibility, and inclusion team were invited to write their own ten simple rules—the rules proposed at that stage are now included in the Supplementary Information.
    We reached a final list by consensus, not popularity. All the other members of the organizing team and the people who helped during the conference (designated helpers) were invited to give their feedback to the resulting list to check for omissions and errors and suggest editions to the text.
    We also sent the manuscript to friendly reviewers who have the experience of attending or organizing conferences. 
    These rules, or their preliminary versions, were shared with 54 people; we received feedback from $\sim$40\%.
    The rules in this manuscript result from this iterative process, discussions with and contributions from all the coauthors, fueled by personal experiences, and extensive literature review.
    A brief description of this process is now stated in lines \ref{methods-a}-\ref{methods-b} of the current version of the manuscript.
    We kept the methodology as brief as possible to focus on the rules instead of our team or conference. 
    
    The coauthors are from diverse backgrounds, career stages, and different continents. They have experienced inequalities in conferences, from different perspectives and levels of inclusion/exclusion and privilege, as stated in lines \ref{background-a}-\ref{background-b}.
    The manuscript summarizes our work together and is strongly supported by the awareness and experience of organizing a global conference that aims to be a kind, inclusive, and accessible experience for as many people as possible.
\end{Reply}

3. In Rule 1, though gender or ethnic diversity has been highlighted, geographical diversity needs to be discussed in detail. There is significant "north-south" divide in scientific research - with several of institutions as well as researchers from LMICs or global south not being given appropriate representation on conference committees as well as on speakers diversity.

\begin{Reply}

   On a global scale, there is an imbalance associated with what could be considered a `north-south' divide, though the definition of north and south may not be clear. We agree that coming from a lower income country hinders participation in conferences. For that reason, we offer an example of registration rates based on country of origin and career status (https://spcanelon.github.io/useR2021-cost-conversion-tool/). 
   On the other hand, while the manuscript is heavily based on our experience organizing a global conference—which is why there are some global examples—the intention is to make the rules general and flexible enough to be useful for local or regional conferences around the world. 
   For that reason, we explicitly mention in Rule 1 that \textit{Lack of diversity will vary across areas of knowledge and depend on the geographic context of the conference} (line \ref{indicators-1}). 

\end{Reply}

4. In Rule 2, safe and inclusive environments, I find little attention to "racist" aberrations or incidents - and how they should be mitigated and when reported be appropriately addressed with fairness and justice.
Several speakers from ethnically diverse backgrounds face racism, exclusion or apathy at such conferences. This need to be discussed.

\begin{Reply}

    Racist acts should be unacceptable behavior in any conference, but we did not mention any example of offenses (including expressions of racism, sexism, ableism, and others). They should be included explicitly in the code of conduct of each conference, as well as the mechanisms for reporting and addressing those reports.
    To further address this comment, we made explicit the link between the diversity statement and the code of conduct, to highlight that all dimensions of diversity should be respected (lines \ref{unacceptable-1} to \ref{unacceptable-2}):
    \textit{The CoC is a living document meant to keep the community safe and should state clearly the behaviors that are deemed unacceptable by the community, the consequences for engaging in such behaviors, and the way to report violations. Bear in mind that every dimension of diversity can be a target of unacceptable behavior. The CoC should help honor the vision expressed in the diversity statement.}
\end{Reply}

5. In Rule 4, vis a vis bias, bias against people with disability and ethnic minorities should also be highlighted.

\begin{Reply}
   
    We had mentioned [perceived] ethnicity as one of the aspects of diversity that may be inferred (correctly or incorrectly) via written communication and be a source of unconscious bias  (line \ref{bias}, \textit{`their perceived origin, ethnicity, gender, or native language'}). 
    
    Conscious and unconscious bias can also operate when the submitted work is not written, but rather in video form. We added a short mention and citation about how video abstracts can be exclusionary (Spitschan et al 2021) by adding barriers to people with visible disabilities, diverse body types, non-native language speakers, or less resources to create videos (lines \ref{video-1} to \ref{video-2}).
    
    Spitschan M, Yehudi Y, Acion L. Video Grant Proposals Could Be Exclusionary. 
    Nature. 2021; 592(7852):26–26. doi:10.1038/d41586-021-00862-x.

    We thank the reviewer for this comment that led us to the current addition.
\end{Reply}

6. In Rule 8, inclusive communication strategy - it is important that the speakers from diverse backgrounds, especially from ethnic minorities etc are highlighted or given prominence in social media strategy or communication. Often, these speakers face relatively less focus. This may be discussed.

\begin{Reply}

    We added the following text to the manuscript (lines \ref{promotion-1} to \ref{promotion-2}) to address this comment: 
    \textit{During the conference, pay special attention to promoting the sessions led, chaired, and presented by people in minoritized groups, multilingual sessions, and the diversified thematic sessions (see \textbf{Rule 4}) to give them the same visibility as all other sessions.}
\end{Reply}

7. In Conclusion, suggest also adding that issues of diversity and inclusion have come to fore even more during COVID-19 as we witness exacerbation of inequities disadvantaging those from vulnerable backgrounds (see https://pubmed.ncbi.nlm.nih.gov/33343410/PMID: 33343410)

\begin{Reply}

    We appreciate the suggestion and agree that the COVID-19 pandemic contributed to the exacerbation of inequity. 
    We would like our manuscript to help inspire and guide the organization of more inclusive conferences across the world, well beyond the pandemic. To ensure this broad applicability, we minimized mentions of the pandemic to only when deemed necessary: to give context to the conference we organized, and to talk about virtual conference formats that prevailed during the last two years.  


    Once again, we thank all the reviewers for their comments and careful consideration.  The feedback has helped clarify the language and the message.
\end{Reply}


\end{document}