Migrating Legacy Projects

Migrating old projects to a new project setup and the python-toolbox can be tedious and annoying. This guide will try to provide some basic steps and guidance to simplify this process.

What a Project Should Have Before You Migrate

• The project configuration should be pyproject.toml based. [Required]

• The project should be poetry based. [Required]

• Dependencies should point only to officially published dependencies on PyPI (no git references or similar) [Required].

• The project documentation should be sphinx based [Required].

• Automated tasks within the project should be available as Python code or Nox tasks. [Helpful]

Hint

If you are interested or want to get a good idea of what a bare standard project based on the toolbox should have, excluding the workflows, it is always good to have a look at the cookiecutter template provided by the toolbox. If the template parts are confusing, you can generate an “empty” project based on it. For details, refer to the Create a New Project with Exasol-Toolbox Support section.

Iterative Migration Guide

Ensure you comply with the basic requirements for your project. Follow these steps for a smooth migration process.

1. Introduce Nox

As a first step, it is generally advisable to introduce Nox as a task runner if it is not already in use. Since the python-toolbox uses Nox as its foundation, this simplifies the migration to the toolbox and enhances the user’s understanding of Nox.

2. Introduce the Standard Nox Tasks

This can be done incrementally for different checks of your project, such as project, test, linting, documentation, and other tasks. As the tasks can be split into roughly five groups, it likely makes sense to integrate at least one group at a time. For more details on the current tasks and groups, refer to: 7. Go 🥜.

Overloading a Task

In certain cases, it may not be trivial to use the nox task defined within the python-toolbox. In those cases, overloads can be used to provide a smoother or faster transition or to cope with project-specific constraints or needs.

For example, if test execution isn’t performed in the standard way (e.g., pytest test/unit, pytest test/integration, pytest test).

Warning

While overwriting can be handy and seem like the simplest way to integrate with the toolbox, please take care when making this decision. In the long run, we want to harmonize projects, their builds, execution, testing, and reporting, etc. If you create an overwrite, you are likely to run into troubles later on, e.g., when reporting is required, because you may lack specific artifacts a specific task was creating, etc.

So while overwriting can be a good temporary solution, for the long term, it should be considered where to add or use a configuration point within the toolbox to make the standard task work for all projects.

Potential options here are:

• Add additional plugin extension points to the toolbox

• Implement functionality for an existing extension point in the python-toolbox

• Add additional configuration parameters in noxconfig.py or pyproject.toml

• Add additional parameterization to tasks

• pytest plugins can take care of a specific test preparation need, e.g., pytest-backend

import nox

# imports all nox task provided by the toolbox
from exasol.toolbox.nox.tasks import *  # pylint: disable=wildcard-import disable=unused-wildcard-import

# default actions to be run if nothing is explicitly specified with the -s option
nox.options.sessions = ["project:fix"]

@nox.session(name="project:fix", python=False)
def my_project_fix_overwrite(session) -> None:
    """Runs all automated fixes on the code base"""

    # ATTENTION:
    # In cases where it is reasonable to use "internal" functions, please do those imports
    # within the function to keep them isolated and simplify future removal or replacement.
    from exasol.toolbox.nox._shared import python_files

    py_files = [f"{file}" for file in python_files(PROJECT_CONFIG.root)]
    print("The original 'project:fix' task has been taken hostage by this overwrite")
    print("Files:\n{files}".format(files="\n".join(py_files))

3. Establish a Baseline

Configure code quality settings in the pyproject.toml file to establish a baseline for your project. If necessary, create tickets for further improvements, especially if major parts of your code require suppression, e.g., in the mypy configuration.

:code:`pyproject.toml` sections to include/consider:

• [tool.coverage.run]

• [tool.coverage.report]

• [tool.black]

• [tool.isort]

• [tool.pylint.format]

• [[tool.mypy.overrides]]

Example

# Adjust this section if you want fine-grained control
# over what is considered for code coverage
[tool.coverage.run]
relative_files = true
source = [
    "exasol",
]

# Adjust this section to define the minimum required
# code coverage for your project
[tool.coverage.report]
fail_under = 15


# Adjust control maximum line length in your project
#
# NOTE:
# As a rule of thumb, you should not exceed 120 characters,
# because overly long lines usually accompany higher cyclomatic complexity,
# as complex functions tend to shift right.
[tool.black]
line-length = 88
include = "\\.pyi?$"


# Adjust to modify the behavior of import sorting
[tool.isort]
profile = "black"
force_grid_wrap = 2


# Adjust to define the minimum linting score considered acceptable for your project
[tool.pylint.master]
fail-under = 7.5

# Maximum line length should match what is configured for black.
# Additionally, a maximum module size can be defined here.
[tool.pylint.format]
max-line-length = 88
max-module-lines = 800


# Configure exceptions for the type checker
[[tool.mypy.overrides]]
module = [
    "test.unit.*",
    "test.integration.*",
]
ignore_errors = true

4. Introduce GitHub Workflows

Install the GitHub workflows provided by the python-toolbox for futher details refer to the section GitHub Workflows.

Attention

This is just guidance. If you have a good understanding of the standard project setup, technologies, and tools used, feel free to diverge at any point or exercise your own judgment.

Migration Progess

Could be tracked in a format and based on the information listed in the real life example bellow.

Hint

This table does not provide any information about the specific python-toolbox used in the respective projects.

Migration Progress

Project	pyproject.toml	poetry	PyPI	Sphinx Docs	nox	toolbox-tasks	toolbox-workflows
python-toolbox	✓	✓	✓	✓	✓	✓	✓
error-reporting-python	✓	✓	✓	✓	✓	✓	✓
pyexasol	✓	✓	✓	✗	✓	✗	✗
sqlalchemy-exasol	✓	✓	✓	✓	✓	✗	✗
bucketfs-python	✓	✓	✓	✓	✓	✓	✓/✗ partialy
ITDE	✓	✓	✓	✓	✓	✓/✗ partialy	✓/✗ partialy
schemas	✓	✓	✗	✗	✗	✗	✗
pytest-plugins	✓	✓	✓	✓/✗ partialy	✓	✓/✗ partialy	✗
harlequin-exasol	✓	✓	✓	✓	✓	✓	✗

Legend

Column	Description
Project	Name of the project
pyproject.toml	Project configuration and setup is pyproject.toml based
poetry	Project configuration and build is Poetry based
PYPI	Project can be build and published to PyPi
Sphinx Docs	The project doumentation is Sphinx based
nox	The projects automated tasks are executed using the Nox task runner
toolbox-tasks	All nox tasks provided by the python-toolbox are available and fully functional
toolbox-workflows	All GitHub Workflows provided by the python-toolbox are available and fully functional