Skip to main content

Performance testing

To test out generation speeds and to provide some utilities to aid profiling Tuist a few auxiliary standalone tools are available.

  • fixturegen: A tool to generate large fixtures
  • tuistbench: A tool to benchmark Tuist

Those tools are located within the projects/ directory.

Benchmarking#

As a convenience to automate the benchmarking process which entails leveraging several tools, a rake task is included with Tuist.

./fourier benchmark

This benchmarks the current branch's version of Tuist against the latest published release using the tools described below.

Fixture Generator#

fixturegen allows generating large fixtures. For example it can generate a workspace with 10 projects, each project with 10 targets, and each target with 500 source files!

Example:

./fourier fixture --projects 100 --targets 10 --sources 500

Generating those large fixtures can be helpful in profiling Tuist and identifying any hot spots that may otherwise go unnoticed when generating smaller fixtures during development.

Tuist Benchmark#

tuistbench has a few modes of operation:

  • Measure the generation time of one or more fixtures
  • Benchmark two Tuist binaries' generation time of one or more fixtures

The benchmark mode can provide a general idea of how changes impact generation time. For example benchmarking a pull request against master or the latest release.

The results are averaged from several cold and warm runs where:

  • cold: Is a generation from a clean slate (no xcodeproj files exist)
  • warm: Is a re-generation (xcodeproj files already exist)

Here are some example outputs from tuistbench.

Measurement (single fixture):

Console format:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     --fixture /path/to/fixtures/ios_app_with_tests
Fixture       : ios_app_with_testsRuns          : 5Result    - cold : 0.72s    - warm : 0.74s

Markdown format:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     --fixture /path/to/fixtures/ios_app_with_tests \     --format markdown
FixtureColdWarm
ios_app_with_tests0.72s0.72s

Benchmark (single fixture):

Console format:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     --reference-binary $(which tuist) \     --fixture /path/to/fixtures/ios_app_with_tests
Fixture       : ios_app_with_testsRuns          : 5Result    - cold : 0.79s  vs  0.80s (≈)    - warm : 0.75s  vs  0.79s (⬇︎ 0.04s 5.63%)

Markdown format:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     --reference-binary $(which tuist) \     --fixture /path/to/fixtures/ios_app_with_tests \     --format markdown
FixtureNewOldDelta
iosapp_with_tests (cold)_0.73s0.79s⬇︎ 7.92%
iosapp_with_tests (warm)_0.79s0.79s

Benchmark (multiple fixtures):

A fixture list json file is needed to specify multiple fixtures, here's an example:

{  "paths": [    "/path/to/fixtures/ios_app_with_tests",    "/path/to/fixtures/ios_app_with_helpers"  ]}

Console:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     --reference-binary $(which tuist) \     --fixture-list fixtures.json
Fixture       : ios_app_with_testsRuns          : 5Result    - cold : 0.79s  vs  0.80s (≈)    - warm : 0.75s  vs  0.79s (⬇︎ 0.04s 5.63%)

Fixture       : ios_app_with_carthage_frameworksRuns          : 5Result    - cold : 0.78s  vs  0.86s (⬇︎ 0.08s 8.90%)    - warm : 0.76s  vs  0.80s (⬇︎ 0.04s 5.05%)

Fixture       : ios_app_with_helpersRuns          : 5Result    - cold : 2.24s  vs  2.37s (⬇︎ 0.12s 5.18%)    - warm : 2.03s  vs  2.11s (⬇︎ 0.07s 3.55%)

Markdown:

swift run tuistbench \     --binary /path/to/tuist/.build/release/tuist \     -reference-binary $(which tuist) \     --fixture-list fixtures.json \     --format markdown
FixtureNewOldDelta
iosapp_with_tests (cold)_0.73s0.79s⬇︎ 7.92%
iosapp_with_tests (warm)_0.79s0.79s
iosapp_with_carthage_frameworks (cold)_0.79s0.85s⬇︎ 7.36%
iosapp_with_carthage_frameworks (warm)_0.77s0.81s⬇︎ 5.26%
iosapp_with_helpers (cold)_2.29s2.43s⬇︎ 5.80%
iosapp_with_helpers (warm)_1.97s2.15s⬇︎ 8.05%