Drake

The benchmark consists of a scene with one or more spheres and one or more cameras above the spheres, looking down at the spheres.

If there are multiple spheres, they are positioned in a regular grid placed at a uniform height above the ground plane. Increasing the number of spheres provides an approximate measure of how the renderer performs with increased scene complexity.

If there are multiple cameras, they are all at the same position, looking in the same direction, with the same intrinsic properties. In other words, each should produce the same output image. This provides a measure of the scalability as a simulation includes an increasing number of cameras.

The output image can be configured to an arbitrary size; larger images take more rendering time.

Warm-starting RenderEngine Implementations

Some RenderEngine implementations may defer some of the initialization work until the rendering API is actually invoked (e.g., RenderEngineVtk). In these cases, the first rendering will be misleadingly expensive. The benchmark "warm starts" each render engine under evaluation by performing two renderings outside of the timed loop. This allows the benchmark to better report the expected results in the engine's "steady state".

When comparing these benchmark results with observed performance in applications, remember that the first and possibly second renderings may deceptively impact any analysis of average render performance.

Benchmarks

The benchmarks have been configured to examine the following characteristics of the included RenderEngine implementations:

We examine those same properties for all three image types: color, depth, and label.

Running the benchmark

The benchmark can be executed as:

bazel run //geometry/benchmarking:render_benchmark

The output will be something akin to the following (although not identical; lines have been removed for brevity):

Run on (12 X 4400 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 12288K (x1)
Load Average: 9.75, 4.65, 3.26
------------------------------------------------------------------------------------ // NOLINT(*)
Benchmark Time CPU Iterations // NOLINT(*)
------------------------------------------------------------------------------------ // NOLINT(*)
RenderBenchmark/VtkColor/1/1/640/480 0.641 ms 0.641 ms 1026 // NOLINT(*)
RenderBenchmark/VtkColor/12/1/640/480 0.843 ms 0.843 ms 882 // NOLINT(*)
RenderBenchmark/VtkColor/120/1/640/480 2.58 ms 2.58 ms 280 // NOLINT(*)
RenderBenchmark/VtkColor/240/1/640/480 5.30 ms 5.30 ms 100 // NOLINT(*)
RenderBenchmark/VtkColor/480/1/640/480 9.10 ms 9.10 ms 79 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/640/480 23.4 ms 23.4 ms 30 // NOLINT(*)
RenderBenchmark/VtkColor/1/10/640/480 6.36 ms 6.36 ms 117 // NOLINT(*)
RenderBenchmark/VtkColor/1200/10/640/480 179 ms 179 ms 4 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/320/240 0.267 ms 0.267 ms 2318 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/1280/960 2.39 ms 2.39 ms 317 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/2560/1920 11.5 ms 11.5 ms 65 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/320/240 24.1 ms 24.0 ms 30 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/1280/960 27.0 ms 26.9 ms 26 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/2560/1920 39.8 ms 39.7 ms 19 // NOLINT(*)
RenderBenchmark/GlColor/1/1/640/480 0.523 ms 0.480 ms 1382 // NOLINT(*)
RenderBenchmark/GlColor/12/1/640/480 0.553 ms 0.511 ms 1236 // NOLINT(*)
RenderBenchmark/GlColor/120/1/640/480 0.833 ms 0.790 ms 884 // NOLINT(*)
RenderBenchmark/GlColor/240/1/640/480 1.20 ms 1.15 ms 591 // NOLINT(*)
RenderBenchmark/GlColor/480/1/640/480 1.87 ms 1.83 ms 384 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/640/480 3.76 ms 3.72 ms 188 // NOLINT(*)
RenderBenchmark/GlColor/1/10/640/480 4.98 ms 4.56 ms 137 // NOLINT(*)
RenderBenchmark/GlColor/1200/10/640/480 37.1 ms 36.6 ms 19 // NOLINT(*)
RenderBenchmark/GlColor/1/1/320/240 0.151 ms 0.147 ms 4830 // NOLINT(*)
RenderBenchmark/GlColor/1/1/1280/960 1.93 ms 1.88 ms 335 // NOLINT(*)
RenderBenchmark/GlColor/1/1/2560/1920 9.14 ms 9.08 ms 72 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/320/240 3.46 ms 3.42 ms 199 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/1280/960 5.29 ms 5.25 ms 127 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/2560/1920 12.5 ms 12.5 ms 57 // NOLINT(*)
RenderBenchmark/VtkDepth/1/1/640/480 1.89 ms 1.89 ms 391 // NOLINT(*)
RenderBenchmark/GlDepth/1/1/640/480 0.486 ms 0.486 ms 1578 // NOLINT(*)
RenderBenchmark/VtkLabel/1/1/640/480 1.35 ms 1.35 ms 513 // NOLINT(*)
RenderBenchmark/GlLabel/1/1/640/480 1.34 ms 1.30 ms 521 // NOLINT(*)

Additional configuration is possible via the following flags:

For example:

bazel run //geometry/benchmarking:render_benchmark -- \
--save_image_path="/tmp" --show_window=true

Interpreting the benchmark

Each line in the table represents a particular configuration of the benchmark parameters of the form:

RenderBenchmark/TestName/sphere_count/camera_count/image_width/image_height

The Time and CPU columns are measures of the average time it took to create a single frame for all the specified cameras. The Iterations indicates how often the action was performed to compute the average value. For more information see the google benchmark documentation.

Now we can analyze the example output and draw some example inferences (not a complete set of valid inferences):