Continuous integration at scale is costly but essential to modern software development. Different test optimization techniques including test selection and prioritization have been proposed to reduce the cost. Test batching is an effective alternative, but often overlooked test optimization technique. In this study, we evaluate the impact of parallelization by varying the number of machines on test batching and propose two new test batching approaches.

We use the default TestAll approach as a baseline to examine the impact of parallelism and machine count on feedback time, which is the time taken for test verdicts to be available for each change. Additionally, we re-evaluate the ConstantBatching technique and introduce the BatchAll algorithm, which adjusts batch size based on the remaining changes in the queue. We also propose TestCaseBatching, which allows new builds to be added to a batch before all tests are executed, accelerating continuous integration. The evaluations are conducted using test results from CompanyA, a proprietary application, and 276 million test outcomes from the open-source Chrome project. Our assessments focus on feedback time and execution reduction, and we provide access to our scripts and data for the Chrome project.

The results reveal a non-linear impact of test parallelization on feedback time, as each test delay compounds across the entire test queue. ConstantBatching, with a batch size of 4, utilizes up to 72% fewer machines to maintain the actual average feedback time and provides a constant execution reduction of up to 75%. Similarly, BatchAll maintains the actual average feedback time with up to 91% fewer machines and exhibits variable execution reduction of up to 99%. TestCaseBatching holds the line of the actual average feedback time with up to 81% fewer machines and demonstrates variable execution reduction of up to 67%. We recommend practitioners use BatchAll and TestCaseBatching as both can reduce the number of machines needed for testing. It is crucial to first examine historical data and determine the threshold at which increasing the number of machines minimally affects feedback time to ensure efficient testing and resource utilization.