Automate git bisect for Xcode regressions on a cloud Mac

A UI test has been failing consistently since last week, but dozens of commits have landed recently. Checking out each one, opening Xcode, and running the test manually is both slow and prone to missed environment differences. A more effective approach is to turn “did this commit introduce the target failure?” into a repeatable command, then let git bisect perform the binary search. With 64 candidate commits, only about 6 rounds are needed in theory. The real challenge is not the binary search itself, but making every result trustworthy.

Define the regression as a machine-verifiable result

Before starting, identify at least two boundaries: one commit confirmed to be good and one confirmed to be bad. Do not settle for a vague description such as “the page is broken.” Narrow the regression to a single observable outcome, such as a specific XCTest assertion failing, a designated Scheme failing to archive, or a fixed input producing the wrong output.

Keep all of the following variables fixed in the test criteria:

• Scheme, Configuration, and test target
• Xcode selection path and dependency lockfiles
• Simulator model, OS version, language, and time zone
• Test data, network dependencies, and execution account
• The single source of evidence used to classify a commit as good or bad

git bisect faithfully amplifies errors in the test harness. If one intermittent failure is classified as a bad commit, the search may complete successfully yet still produce a completely incorrect conclusion.

First, run the same command twice manually on both the good and bad commits. If the results are not reproducible, fix the test isolation before starting the bisect.

Isolate the execution environment for every candidate commit

When running on a G-Mini cloud Mac, use a dedicated working copy rather than a directory where a developer is actively making changes. The bisect process switches commits frequently, and untracked files, generated configuration, or shared caches can all contaminate the result.

Run git status --porcelain before starting; it must produce no output. Create and boot the simulator in advance, then store its UDID in an environment variable. Do not make the test harness select “any available device” by name at runtime. Duplicate device names and OS upgrades can silently change the actual destination.

Write a three-state Xcode test harness

git bisect run recognizes more than success and failure. Exit code 0 means good, 1 through 127 mean bad, and 125 means the current commit cannot be evaluated and should be skipped. This third state is especially important for older projects: an earlier commit may no longer build with current dependencies, but that does not necessarily mean it contains the regression under investigation.

The following script first checks whether the project can build, then runs only the target test. A baseline build failure is classified as untestable; only an explicit test failure is classified as bad.

#!/bin/zsh
set -u

: "${DESTINATION_ID:?Set DESTINATION_ID first}"

sha="$(git rev-parse --short HEAD)"
root="${TMPDIR:-/tmp}/xcode-bisect"
derived="$root/derived-$sha"
result="$root/result-$sha.xcresult"
log="$root/test-$sha.log"

mkdir -p "$root"
rm -rf "$result"

xcodebuild \
  -project App.xcodeproj \
  -scheme App \
  -configuration Debug \
  -destination "platform=iOS Simulator,id=$DESTINATION_ID" \
  -derivedDataPath "$derived" \
  build >"$root/build-$sha.log" 2>&1

if [[ $? -ne 0 ]]; then
  exit 125
fi

xcodebuild \
  -project App.xcodeproj \
  -scheme App \
  -destination "platform=iOS Simulator,id=$DESTINATION_ID" \
  -derivedDataPath "$derived" \
  -resultBundlePath "$result" \
  -only-testing:AppTests/CheckoutReducerTests/testExpiredCart \
  test >"$log" 2>&1

status=$?

if [[ $status -eq 0 ]]; then
  exit 0
fi

if grep -q "TEST FAILED" "$log"; then
  exit 1
fi

exit 125

Adjust the classification for the failure type

If you are investigating a build regression, a build failure should return 1 rather than 125. If you are investigating a behavioral regression, infrastructure problems such as dependency download failures, simulator service errors, or disk write failures must be skipped. Do not classify every nonzero xcodebuild exit code as the target regression.

Run the bisect and verify the first bad commit

Once the script is ready, record the current branch and workspace state, then run:

chmod +x ./scripts/bisect-xcode.sh
export DESTINATION_ID="固定的模拟器UDID"

git bisect start
git bisect bad BAD_COMMIT
git bisect good GOOD_COMMIT
git bisect run ./scripts/bisect-xcode.sh

When the search finishes, Git will identify the first bad commit. Do not close the issue immediately. Run the script manually on both that commit and its parent, then inspect the saved build logs and .xcresult files. Also review the commit diff to confirm that it can explain the observed behavior rather than merely triggering a different failure.

When finished, run git bisect reset to return to the original branch. If the bisect generated a large amount of DerivedData, clean it up after verification. Do not delete result bundles prematurely during the investigation, or you will lose the evidence behind the classifications.

Handle flaky tests and gaps in project history

Use majority voting for flaky tests

If a single run is unreliable, have the harness run the test three times and return 1 only when the same target failure occurs at least twice. If the three results conflict, return 125. This increases execution time, but it is still cheaper than investigating the wrong commit. Reset the test data before every repeated run so that state from the previous run cannot affect the next one.

Narrow the boundaries when too many commits are skipped

A large number of 125 results can prevent Git from identifying a unique commit. Common causes include historical changes to the project format, dependency management approach, or test name. In that case, move the search boundary to after the migration, or add compatibility branches for the older directory structure. Do not let the script modify the source code under test automatically.

The final investigation record should include the good commit, bad commit, first bad commit, test harness version, simulator UDID, Xcode version, and result bundle paths. This makes the result reproducible by another engineer and allows the same harness to be reused unchanged for regression verification after the fix is committed.

Frequently asked questions