Performance Test#

Attention

For the performance tests, a Docker container with a test database needs to be started. This can be done with poetry run -- nox -s db:start.

As these tests are meant to measure the performance of functions, the tests are expected to take a bit to set up and execute. If you are running the tests for the first time, it is recommend that you modify the throughput data values to a smaller value; this can be done by modifying BenchmarkSpecifications.target_data_size.

To execute the performance tests, you need to additionally execute:

poetry install --with performance

This installs pytest-benchmark <https://pypi.org/project/pytest-benchmark/>__, which provides a pytest fixture benchmark to re-run tests a specified number of times, to capture how long each execution took to run, and to calculate various statistical values for comparative purposes.

Note

pytest-benchmark is not included in the dev dependencies due to an odd correlation observed when executing the integration tests. This was observed with pytest-benchmark version 5.1.0 and its dependency py-cpuinfo 9.0.0. Namely, when executing the integration tests with Python 3.11, we saw that the tests in transaction_test.py consistently failed. This was not observed for other Python versions and could not be correlated with other changes at the time of investigation.

Updating the Benchmark JSON File#

To ensure that PyExasol’s functional performance does not degrade over time. If the comparison fails, a developer should scrutinize the changes to determine the cause and, if deemed relevant, update the test/performance/.benchmarks/0001_performance.json with results saved from the CI run (i.e. from the relevant artifact):

  • Is it a new test?

    • Then, add it to the JSON file.

  • Was a test removed?

    • If we meant to remove it, remove it from the JSON file.

  • What change made that impacted the tests?

    • If it’s a change we intended and cannot improve upon, then alter the corresponding benchmark results in the JSON file.