Call for Papers

We invite high-quality submissions, from both industry and academia, describing original and unpublished results of theoretical, empirical, conceptual, and experimental software engineering research.

NEW THIS YEAR: major revisions allowed! The main novelty of this year’s review process is that the initial output can be accept, reject or major revision. In case a paper is deemed publishable upon major revision, authors are granted 8 weeks to perform major revisions, which might include additional experiments or new analyses of existing results; major rewriting of algorithms and explanations; clarifications, better scoping, and improved motivations. The same reviewers who requested major revisions will then assess whether the revised submission satisfies their requests adequately.

NEW THIS YEAR: research methods declaration! This year, in addition to declaring the topics which are relevant for their submissions, authors will be asked to declare the research methods employed in their submissions. This will enable us to ensure reviewer expertise both for research methods and topics. For full definitions of the research methods, see the SIGSOFT Empirical Standards.

NEW THIS YEAR: adoption of SIGSOFT Open Science Policy! This year, ESEC/FSE has adopted the SIGSOFT Open Science Policy, and we encourage authors to provide a replication package. For authors, this means providing a supporting statement on the availability of a replication package (or lack thereof) in their submitted papers in a section named Data Availability after the Conclusion section. See more details below.

Contributions should describe innovative and significant original research. Papers describing groundbreaking approaches to emerging problems are also welcome, as well as replication papers. Submissions that facilitate reproducibility by using available datasets or making the described tools and datasets publicly available are especially encouraged. For a list of specific topics of interest, please see the end of this call.

How to Submit

At the time of submission, all papers must conform to the ESEC/FSE 2023 Format and Submission Guidelines, and must not exceed 10 pages for all text and figures plus 2 pages for references. All submissions must be in English and in PDF format. Submissions that do not comply with the above instructions will be desk rejected without review. Papers must be submitted electronically through the ESEC/FSE submission site:

https://esecfse2023.hotcrp.com/

Each submission will be reviewed by at least three members of the program committee. When the initial output of the three reviews is major revision, authors will have an opportunity to address the reviewers’ requests during an 8 week major revision period. The revised submission must be accompanied by a response letter, where the authors explain how they addressed each concern expressed by the reviewers.

Submissions will be evaluated on the basis of originality, importance of contribution, soundness, evaluation (if relevant), quality of presentation, and appropriate comparison to related work. Some papers may have more than three reviews, as PC chairs may solicit additional reviews based on factors such as reviewer expertise and strong disagreement between reviewers. The program committee as a whole will make final decisions about which submissions to accept for presentation at the conference.

Double-Anonymous Review Process

In order to ensure the fairness of the reviewing process, the ESEC/FSE 2023 Research Papers Track will employ a double-anonymous review process, where external reviewers do not know the identity of authors, and authors do not know the identity of external reviewers. The papers submitted must not reveal the authors’ identities in any way:

• Authors should leave out author names and affiliations from the body of their submission.
• Authors should ensure that any citation to related work by themselves is written in third person, that is, “the prior work of XYZ” as opposed to “our prior work”.
• Authors should not include URLs to author-revealing sites (tools, datasets). Authors are still encouraged to follow open science principles and submit replication packages, see more details on the open science policy below.
• Authors should anonymize author-revealing company names but instead provide general characteristics of the organisations involved needed to understand the context of the paper.
• Authors should ensure that paper acknowledgements do not reveal the origin of their work.

The double-anonymous process used this year is “heavy”, i.e., the paper anonymity will be maintained during all reviewing and discussion periods. In case of major revision, authors must therefore maintain anonymity in their response letter and must provide no additional information that could be author-revealing.

To facilitate double-anonymous reviewing, we recommend the authors to postpone publishing their submitted work on arXiv or similar sites until after the notification. If the authors have uploaded to arXiv or similar, they should avoid specifying that the manuscript was submitted to ESEC/FSE 2023.

Authors with further questions on double-anonymous reviewing are encouraged to contact the program chairs by email. Papers that do not comply with the double-anonymous review process will be desk-rejected.

Submission Policies

Papers submitted for consideration to ESEC/FSE should not have been already published elsewhere and should not be under review or submitted for review elsewhere during the reviewing period. Specifically, authors are required to adhere to the ACM Policy and Procedures on Plagiarism and the ACM Policy on Prior Publication and Simultaneous Submissions.

To prevent double submissions, the chairs might compare the submissions with related conferences that have overlapping review periods. The double submission restriction applies only to refereed journals and conferences, not to unrefereed forums (e.g. arXiv.org). To check for plagiarism issues, the chairs might use external plagiarism detection software.

All publications are subject to the ACM Author Representations policy.

By submitting your article to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all ACM Publications Policies, including ACM’s new Publications Policy on Research Involving Human Participants and Subjects.

Alleged violations to any of the above policies will be reported to ACM for further investigation and may result in a full retraction of your paper, in addition to other potential penalties, as per the ACM Publications Policies.

Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process if your paper is accepted. ACM has been involved in ORCID from the start and they have recently made a commitment to collect ORCID IDs from all of published authors. The collection process has started and will roll out as a requirement throughout 2022. ACM is committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.

Important Dates

All dates are 23:59:59 AoE (UTC-12h)

• Paper registration: 26 January, 2023 (to register a paper, only the paper title, author list and some additional metadata are required)
• Full paper submission: 2 February, 2023
• Initial notification: 4 May, 2023
• Revised manuscript submissions (major revisions only): 29 June, 2023
• Final notification for major revisions: 27 July, 2023
• Camera ready: 24 August, 2023

NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work.

Open Science Policy

The research track of ESEC/FSE has introduced an open science policy. Openness in science is key to fostering scientific progress via transparency, reproducibility, and replicability. The steering principle is that all research results should be accessible to the public, if possible, and that empirical studies should be reproducible. In particular, we actively support the adoption of open data and open source principles and encourage all contributing authors to disclose (anonymized and curated) data to increase reproducibility and replicability.

Upon submission to the research track, authors are asked to make a replication package available to the program committee (via upload of supplemental material or a link to a private or public repository) or to comment on why this is not possible or desirable. Furthermore, authors are asked to indicate whether they intend to make their data publicly available upon acceptance. We ask authors to provide a supporting statement on the availability of a replication package (or lack thereof) in their submitted papers in a section named Data Availability after the Conclusion section. Be careful that such statements continue to maintain author anonymity. For more details, see the ESEC/FSE open science policy.

Authors of accepted papers will be given an opportunity (and encouragement) to submit their data and tools to the separate ESEC/FSE’23 artefact evaluation committee.

Topics of Interest

Topics of interest include, but are not limited to:

• Artificial intelligence and machine learning for software engineering
• Autonomic computing
• Debugging and fault localization
• Dependability, safety, and reliability
• Distributed and collaborative software engineering
• Embedded software, safety-critical systems, and cyber-physical systems
• Empirical software engineering
• Human aspects of software engineering
• Human-computer interaction
• Mining software repositories
• Mobile development
• Model checking
• Model-driven engineering
• Parallel, distributed, and concurrent systems
• Performance engineering
• Program analysis
• Program comprehension
• Program repair
• Program synthesis
• Programming languages
• Recommendation systems
• Requirements engineering
• Search-based software engineering
• Services, components, and cloud
• Software architectures
• Software engineering education
• Software engineering for machine learning and artificial intelligence
• Software evolution
• Software processes
• Software security
• Software testing
• Software traceability
• Symbolic execution
• Tools and environments

FAQ on Review Process: Major Revisions, Open Science Policy, Double-Anonymous Reviewing

Major Revision Process

Q: Why is ESEC/FSE allowing major revisions?

A: SE conferences are currently forced to reject papers that include valuable material, but would need major changes to become acceptable for conference presentation, because major revisions cannot be accommodated in the current review process. By supporting only a binary outcome, conferences force reviewers to decide between rejection and acceptance even in borderline cases that would be better judged after a round of major revision. This can cause additional reviewing burden for the community (the paper is resubmitted to another venue with new reviewers) and inconsistency for the authors (the new reviewers have different opinions). We hope by allowing major revisions to both increase the acceptance rate of ESEC/FSE and to help reduce these current problems with the reviewing process.

For Authors

Q: If my paper receives major revisions, what happens next?

A: The meta-review will clearly and explicitly list all major changes required by the reviewers to make the paper acceptable for publication. Authors of these papers are granted 8 weeks to implement the requested changes. In addition to the revised paper, authors are asked to submit a response letter that explains how each required change was implemented. If any change was not implemented, authors can explain why. The same reviewers will then review the revised paper and make their final (binary) decision. Authors can also choose to withdraw their submission if they wish.

Q: Will major revision become the default decision causing initial acceptance rates to drop?

A: This is not the intention: reviewers are instructed to accept all papers that would have been accepted when major revision was not an available outcome.

For Reviewers

Q: When shall I recommend major revision for a paper?

A: Major revision should not become the default choice for borderline papers and should be used only if without major revisions the paper would be rejected, while a properly done major revision, which addresses the reviewers’ concerns, could make the paper acceptable for publication; if the requested changes are doable in 8 weeks and are implementable within the page limit; if the requested changes are strictly necessary for paper acceptance (i.e., not just nice-to-have features); if the requested changes require recheck (i.e., reviewers cannot trust the authors to implement them directly in the camera ready).

Q: When shall I recommend rejection instead of major revision?

A: Rejection is a more appropriate outcome than major revision if the requested additions/changes are not implementable in 8 weeks; if the contribution is very narrow or not relevant to the SE audience, and it cannot be retargeted in 8 weeks; if the methodology is flawed and cannot be fixed in 8 weeks; if results are unconvincing, the paper does not seem to improve the state of the art much, and new convincing results are unlikely to be available after 8 weeks of further experiments; if the customary benchmark used in the community was ignored and cannot be adopted and compared to in 8 weeks.

Q: When shall I recommend acceptance instead of major revision?

A: We do not want major revision to become the primary pathway for acceptance. We should continue to trust the authors to make minor changes to the submissions in the camera ready version. Acceptance is preferable if the requested additions/changes are nice to have features, not mandatory for the acceptability of the work; if minor improvements of the text are needed; if minor clarifications requested by the reviewers should be incorporated; if important but not critical references should be added and discussed; if discussion of results could be improved, but the current one is already sufficient.

Q: What is the difference between major revision and shepherding?

A: Major revision is not shepherding. While shepherding typically focuses on important but minor changes, which can be specified in an operational way and can be checked quite easily and quickly by reviewers, major revisions require major changes (although doable in 8 weeks), which means the instructions for the authors cannot be completely operational and the check will need to go deeply into the new content delivered by the paper. Hence, while the expectation for shepherded papers is that most of them will be accepted once the requested changes are implemented, this is not necessarily the case with major revisions.

Q: Is there a quota of papers that can have major revision as outcome?

A: As there is no quota for the accepted papers, there is also no quota for major revisions. However, we expect that thanks to major revisions we will be able to eventually accept 10-15% more papers, while keeping the quality bar absolutely unchanged.

Q: What shall I write in the meta-review of a paper with major revision outcome?

A: With the possibility of a major revision outcome, meta-reviews become extremely important. The meta-review should clearly and explicitly list all major changes required by the reviewers to make the paper acceptable for publication. The meta-review should act as a contract between reviewers and authors, such that when all required changes are properly made, the paper is accepted. In this respect, the listed changes should be extremely clear, precise, and implementable.

Review Process

For Authors

Q: Can I withdraw my paper?

A: Yes, papers can be withdrawn at any time using HotCRP.

For Reviewers

Q: The authors have provided a URL to supplemental material. I would like to see the material but I worry they will snoop my IP address and learn my identity. What should I do?

A: Contact the Program Co-Chairs, who will download the material on your behalf and make it available to you.

Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?

A: PC members should do their own reviews, not delegate them to someone else. Please contact the Program Co-Chairs, especially since additional reviewers might have a different set of conflicts of interest.

Open Science Policy

Q: What is the ESEC/FSE 2023 open science policy and how can I follow it?

A: Openness in science is key to fostering scientific progress via transparency, reproducibility, and replicability. Upon submission to the research track, authors are asked to:

• make their data available to the program committee (via upload of supplemental material or a link to an anonymous repository) and provide instructions on how to access this data in the paper; or
• include in the paper an explanation as to why this is not possible or desirable; and
• indicate if they intend to make their data publicly available upon acceptance. This information should be provided in the submitted papers in a section named Data Availability after the Conclusion section. For more details, see the ESEC/FSE open science policy.

Q: How can I upload supplementary material via the HotCRP site and make it anonymous for double-anonymous review?

A: To conform to the double-anonymous policy, please include an anonymized URL. Code and data repositories may be exported to remove version control history, scrubbed of names in comments and metadata, and anonymously uploaded to a sharing site. Instructions are provided in the ESEC/FSE open science policy.

Double-Anonymous Reviewing (DAR)

Q: Why are you using double-anonymous reviewing?

A: Studies have shown that a reviewer’s attitude toward a submission may be affected, even unconsciously, by the identity of the authors.

Q: Do you really think DAR actually works? I suspect reviewers can often guess who the authors are anyway.

A: It is rare for authorship to be guessed correctly, even by expert reviewers, as detailed in this study.

For Authors

Q: What exactly do I have to do to anonymize my paper?

A: Your job is not to make your identity undiscoverable but simply to make it possible for reviewers to evaluate your submission without having to know who you are: omit authors’ names from your title page, and when you cite your own work, refer to it in the third person. Also, be sure not to include any acknowledgements that would give away your identity. You should also avoid revealing the institutional affiliation of authors.

Q: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems. How do I do this?

A: On the submission site, there will be an option to submit supplementary material along with your main paper. You can also share supplementary material in a private or publicly shared repository (preferred). This supplementary material should also be anonymized; it may be viewed by reviewers during the review period, so it should adhere to the same double-anonymous guidelines. See instructions on the ESEC/FSE open science policy.

Q: My submission is based on code available in a public repository. How do I deal with this?

A: Making your code publicly available is not incompatible with double-anonymous reviewing. You can create an anonymized version of the repository and include a new URL that points to the anonymized version of the repository, similar to how you would include supplementary materials to adhere to the Open Science policy. Authors wanting to share GitHub repositories may want to look into using https://anonymous.4open.science/ which is an open source tool that helps you to quickly double-anonymize your repository.

Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?

A: Maybe. The core question is really whether the system is one that, once identified, automatically identifies the author(s) and/or the institution. If the system is widely available, and especially if it has a substantial body of contributors and has been out for a while, then these conditions may not hold (e.g., LLVM or HotSpot), because there would be considerable doubt about authorship. By contrast, a paper on a modification to a proprietary system (e.g., Visual C++, or a research project that has not open-sourced its code) implicitly reveals the identity of the authors or their institution. If naming your system essentially reveals your identity (or institution), then anonymize it. In your submission, point out that the system name has been anonymized. If you have any doubts, please contact the Program Co-Chairs.

Q: I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?

A: No. But we recommend you do not use the same title for your ESEC/FSE submission, so that it is clearly distinguished from the prior paper. In general, there is rarely a good reason to anonymize a citation. When in doubt, contact the Program Co-Chairs.

Q: Am I allowed to post my (non-anonymized) paper on my web page or arXiv?

A: To facilitate double-anonymous reviewing, we recommend the authors to postpone publishing their submitted work on arXiv or similar sites until after the notification. If the authors have uploaded to arXiv or similar, they should avoid specifying that the manuscript was submitted to ESEC/FSE 2023.

Q: Can I give a talk about my work while it is under review? How do I handle social media?

A: We have developed guidelines, described here, to help everyone navigate in the same way the tension between the normal communication of scientific results, which double-anonymous reviewing should not impede, and actions that essentially force potential reviewers to learn the identity of the authors for a submission. Roughly speaking, you may (of course!) discuss work under submission, but you should not broadly advertise your work through media that is likely to reach your reviewers. We acknowledge there are grey areas and trade-offs; we cannot describe every possible scenario.

Things you may do:

• Put your submission on your home page.
• Discuss your work with anyone who is not on the review committees, or with people on the committees with whom you already have a conflict.
• Present your work at professional meetings, job interviews, etc.
• Submit work previously discussed at an informal workshop, previously posted on arXiv or a similar site, previously submitted to a conference not using double-anonymous reviewing, etc.

Things you should not do:

• Contact members of the review committees about your work, or deliberately present your work where you expect them to be.
• Publicize your work on major mailing lists used by the community (because potential reviewers likely read these lists).
• Publicize your work on social media if wide public [re-]propagation is common (e.g., Twitter) and therefore likely to reach potential reviewers. For example, on Facebook, a post with a broad privacy setting (public or all friends) saying, “Whew, ESEC/FSE paper in, time to sleep” is okay, but one describing the work or giving its title is not appropriate. Alternatively, a post to a group including only the colleagues at your institution is fine.

Reviewers will not be asked to recuse themselves from reviewing your paper unless they feel you have gone out of your way to advertise your authorship information to them. If you are unsure about what constitutes “going out of your way”, please contact the Program Co-Chairs.

Q: Will the fact that ESEC/FSE is double-anonymous have an impact on handling conflicts of interest? A: Double-anonymous reviewing does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Authors declare conflicts of interest when submitting their papers using the guidelines in the Call for Papers. Papers will not be assigned to reviewers who have a conflict. Note that you should not declare gratuitous conflicts of interest and the chairs will compare the conflicts declared by the authors with those declared by the reviewers. Papers abusing the system will be desk-rejected.

For Reviewers

Q: What should I do if I learn the authors’ identity? What should I do if a prospective ESEC/FSE author contacts me and asks to visit my institution?

A: If you feel that the authors’ actions are largely aimed at ensuring that potential reviewers know their identity, contact the Program Co-Chairs. Otherwise, you should not treat double-anonymous reviewing differently from other reviewing. In particular, refrain from seeking out information on the authors’ identity, but if you discover it accidentally this will not automatically disqualify you as a reviewer. Use your best judgement.

Q: How do we handle potential conflicts of interest since I cannot see the author names?

A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system.

Q: How should I avoid learning the authors’ identity, if I am using web-search in the process of performing my review?

A: You should make a good-faith effort not to find the authors’ identity during the review period, but if you inadvertently do so, this does not disqualify you from reviewing the paper. As part of the good-faith effort, please turn off Google Scholar auto-notifications. Please do not use search engines with terms like the paper’s title or the name of a new system being discussed. If you need to search for related work you believe exists, do so after completing a preliminary review of the paper.

---

The above guidelines are partly based on the PLDI FAQ on double-anonymous reviewing and the ICSE 2022 guidelines on double-anonymous submissions.