API Reference

exasol.nb_connector

exasol.nb_connector.bfs_utils.put_file(bucket: Bucket, file_path: Path, skip_if_exists: bool = True) → BucketPath

Uploads given file into bucketfs :param bucket: bucket to use :param file_path: local file path to uplaod. File have to exist. :param skip_if_exists: Do not upload if file already present in the bucketfs. :return: Path in the bucketfs.

exasol.nb_connector.cloud_storage.setup_scripts(db_connection: ExaConnection, schema_name: str, bucketfs_jar_path: str)

Perform initialization of scripts for could-storage-extension. :param db_connection: DB connection :param schema_name: name of the schema to be used. :param bucketfs_jar_path: path to cloud-storage-extension jar in BucketFS :return:

exasol.nb_connector.github.retrieve_jar(project: Project, use_local_cache: bool = True, storage_path: Path | None = None) → Path

Returns latest jar file for the project, possibly using local cache.

Parameters:

• project – project to be used

• use_local_cache – should local cache be used or file always retrieved anew

• storage_path – path to be used for downloading. If None, current directory will be used.

Returns:

path to the jar file on the local filesystem

class exasol.nb_connector.ai_lab_config.AILabConfig(value)

An enumeration.

accelerator = 42

aws_access_key_id = 25

aws_region = 24

aws_secret_access_key = 26

bfs_bucket = 17

bfs_connection_name = 44

bfs_encryption = 20

bfs_host_name = 12

bfs_internal_host_name = 14

bfs_internal_port = 15

bfs_model_subdir = 43

bfs_password = 19

bfs_port = 13

bfs_service = 16

bfs_user = 18

cert_vld = 8

client_cert = 10

client_key = 11

db_encryption = 7

db_host_name = 2

db_password = 6

db_port = 3

db_schema = 4

db_user = 5

disk_size = 22

huggingface_token = 23

itde_container = 27

itde_network = 29

itde_volume = 28

mem_size = 21

saas_account_id = 38

saas_database_id = 39

saas_database_name = 40

saas_token = 37

saas_url = 36

sme_aws_bucket = 33

sme_aws_connection = 35

sme_aws_role = 34

storage_backend = 41

te_hf_connection = 30

te_models_cache_dir = 31

trusted_ca = 9

txaie_models_cache_dir = 32

use_itde = 1

class exasol.nb_connector.ai_lab_config.Accelerator(value)

An enumeration.

none = 'none'

nvidia = 'nvidia'

class exasol.nb_connector.ai_lab_config.StorageBackend(value)

An enumeration.

onprem = 1

saas = 2

exasol.nb_connector.connections

exasol.nb_connector.connections.open_pyexasol_connection(conf: Secrets, **kwargs) → ExaConnection

Opens a pyexasol connection using provided configuration parameters. Supports both On-Prem and Saas backends. Does NOT set the default schema, even if it is defined in the configuration.

Any additional parameters can be passed to pyexasol via the kwargs. Parameters in kwargs override the correspondent values in the configuration.

The configuration should provide the following parameters:

On-Prem:

• Server address and port (db_host_name, db_port),

• Client security credentials (db_user, db_password).

Saas:

• SaaS service url (saas_url),

• SaaS account id (saas_account_id),

• Database id or name (saas_database_id or saas_database_name),

• Client security credentials (saas_token).

Optional parameters include:

• Secured comm flag (db_encryption),

• Some of the SSL options (cert_vld, trusted_ca, client_cert).

If the schema is not provided then it should be set explicitly in every SQL statement. For other optional parameters the default settings are as per the pyexasol interface.

exasol.nb_connector.connections.open_bucketfs_bucket(conf: Secrets) → BucketLike

Connects to a BucketFS service using provided configuration parameters. Returns the BucketLike object for the bucket selected in the configuration. Supports both On-Prem and Saas backends.

The configuration should provide the following parameters;

On-Prem:

• Host name and port of the BucketFS service (bfs_host_name or db_host_name, bfs_port),

• Client security credentials (bfs_user, bfs_password).

• Bucket name (bfs_bucket)

Saas:

• SaaS service url (saas_url),

• SaaS account id (saas_account_id),

• Database id or name (saas_database_id or saas_database_name),

• Client security credentials (saas_token).

Optional parameters include:

• Secured comm flag (bfs_encryption), defaults to False.

• Some of the SSL options (cert_vld, trusted_ca).

exasol.nb_connector.connections.open_bucketfs_location(conf: Secrets) → PathLike

Similar to open_buckets_connection, but returns a PathLike interface.

exasol.nb_connector.connections.get_saas_database_id(conf: Secrets) → str

Gets the SaaS database id using the available configuration elements.

exasol.nb_connector.connections.get_backend(conf: Secrets) → StorageBackend

Tries to find which backend was selected in the configuration. If the relevant configuration element is not there - which may be the case if the configuration has been created before the SaaS support was introduced - returns the StorageBackend.onprem.

exasol.nb_connector.connections.get_udf_bucket_path(conf: Secrets) → str

Builds the path of the BucketFS bucket specified in the configuration, as it’s seen in the udf’s file system.

exasol.nb_connector.connections.open_sqlalchemy_connection(conf: Secrets)

Creates an Exasol SQLAlchemy websocket engine using provided configuration parameters. Supports both On-Prem and Saas backends. Sets the default schema if it is defined in the configuration.

The configuration should provide the following parameters:

On-Prem:

• Server address and port (db_host_name, db_port),

• Client security credentials (db_user, db_password).

Saas:

• SaaS service url (saas_url),

• SaaS account id (saas_account_id),

• Database id or name (saas_database_id or saas_database_name),

• Client security credentials (saas_token).

Optional parameters include:

• Secured comm flag (db_encryption).

• Validation of the server’s TLS/SSL certificate by the client (cert_vld).

If the schema is not provided then it should be set explicitly in every SQL statement. For other optional parameters the default settings are as per the Exasol SQLAlchemy interface. Currently, it’s not possible to use a bundle of trusted CAs other than the default. Neither it is possible to set the client TLS/SSL certificate.

exasol.nb_connector.connections.open_ibis_connection(conf: Secrets, **kwargs)

Creates a connection to Ibis with Exasol backend.

The parameters are similar to those of open_pyexasol_connection function. The downstream call signature is also similar to pyexasol, except that the dsn is provided in two separate parts - host and port.

Unlike open_pyexasol_connection, this function sets the default schema if it’s defined in the configuration.

exasol.nb_connector.itde_manager

class exasol.nb_connector.itde_manager.ItdeContainerStatus(value)

An enumeration.

ABSENT = 0

READY = 7

RUNNING = 3

STOPPED = 1

VISIBLE = 5

exasol.nb_connector.itde_manager.bring_itde_up(conf: Secrets, env_info: EnvironmentInfo | None = None) → None

Launches the ITDE environment using its API. Sets hardcoded environment name, and Google name server address. Additionally, can set the following parameters with values collected from the secret store: - database port forwarding, - BucketFS port forwarding, - database memory size (the value is assumed to be the number of gigabytes), - database disk size (the value is assumed to be the number of gigabytes).

Optionally, an existing instance of the DockerDB can be used instead. In this case the EnvironmentInfo object must be provided.

The function connects the current container to the network of the container where the Docker DB is running.

The names of created docker container, docker volume and docker network will be saved in the provided secret store. They will be used by the function that takes down the environment.

The function saves the main AI-Lab configuration parameters, such as the DB and BucketFS connection parameters, in the secret store.

exasol.nb_connector.itde_manager.get_itde_status(conf: Secrets) → ItdeContainerStatus

Checks if the ITDE container exists and ready to be used. In the Docker Edition that means the ITDE is running and the AI-Lab container is connected to its network. In other editions it will just check that the ITDE is running.

Returns the container status.

The name of the container is taken from the provided secret store. If the name cannot be found there the function returns the status ABSENT.

exasol.nb_connector.itde_manager.restart_itde(conf: Secrets) → None

Starts an existing ITDE container if it’s not already running. In the Docker Edition connects the AI-Lab container to the Docker-DB network, unless it’s already connected to it.

For this function to work the container must exist. If it doesn’t a RuntimeError will be raised. Use the get_itde_status function to check if the container exists.

exasol.nb_connector.itde_manager.take_itde_down(conf: Secrets, stop_db: bool = True) → None

Shuts down the ITDE. The names of the docker container, docker volume and docker network are taken from the provided secret store. If the names are not found there no action is taken.

Stopping the DockerDB can be skipped by setting the stop_db parameter to False. This is a useful option in case the DockerDB has been provided externally.

exasol.nb_connector.script_language_container

class exasol.nb_connector.slc.script_language_container.ScriptLanguageContainer(secrets: Secrets, name: str)

Support building different flavors of Exasol Script Language Containers (SLCs) using the SLCT.

Parameter name serves as base of the language alias and a key for the related flavor stored in the Secure Configuration Storage (SCS / secrets / conf). The flavor is used as a template for building the SLC.

If the flavor is missing in the SCS or the SLC Git repository has not been checked out (i.e. cloned) into the checkout_dir, then the constructor will raise an SlcError.

Additionally, the caller needs to ensure, that a flavor with this name is contained in the SLC release specified in variable constants.SLC_RELEASE_TAG.

GITHUB_URL = 'https://github.com/exasol/script-languages-release/releases/tag/10.0.0'

Hyperlink to the GitHub tag which is being used.

property activation_key: str

Returns the language activation string for the uploaded script-language-container. Can be used in ALTER SESSION or ALTER_SYSTEM SQL commands to activate the language of the uploaded script-language-container.

append_custom_conda_packages(conda_packages: list[CondaPackageDefinition])

Appends packages to the custom conda packages file. Note: This method is not idempotent: Multiple calls with the same package definitions will result in duplicated entries.

append_custom_pip_packages(pip_packages: list[PipPackageDefinition])

Appends packages to the custom pip packages file. Note: This method is not idempotent: Multiple calls with the same package definitions will result in duplicated entries.

property checkout_dir: Path

classmethod clean_docker_images(output_path: Path = PosixPath('/tmp/tmpavb0u58l/23cbf1eca7741dfdcd78f9de312267529774cc13')) → None

Deletes all SLC related local docker images. :param output_path: Defines the path where the log files are stored.

classmethod create(secrets: Secrets, name: str, flavor: str, compression_strategy: CompressionStrategy = CompressionStrategy.GZIP) → ScriptLanguageContainer

property custom_conda_file: Path

Returns the path to the custom conda packages file of the flavor

property custom_packages_dir

Returns the path to the custom packages directory of the flavor

property custom_pip_file: Path

Returns the path to the custom pip packages file of the flavor

deploy()

Deploys the current script-languages-container to the database and stores the activation string in the Secure Configuration Storage.

property docker_image_tags: list[str]

Return list of Docker image tags related to the current SLC.

export() → None

Exports the current SLC to the export directory.

export_no_copy() → None

Exports the current SLC to the internal output directory only, without copying to the export directory.

property flavor_path: Path

property language_alias: str

Is case-insensitive.

classmethod list_available_flavors() → list[str]

class exasol.nb_connector.slc.CondaPackageDefinition(pkg, version)

pkg

Alias for field number 0

version

Alias for field number 1

class exasol.nb_connector.slc.PipPackageDefinition(pkg, version)

pkg

Alias for field number 0

version

Alias for field number 1

class exasol.nb_connector.slc.ScriptLanguageContainer(secrets: Secrets, name: str)

Support building different flavors of Exasol Script Language Containers (SLCs) using the SLCT.

Parameter name serves as base of the language alias and a key for the related flavor stored in the Secure Configuration Storage (SCS / secrets / conf). The flavor is used as a template for building the SLC.

If the flavor is missing in the SCS or the SLC Git repository has not been checked out (i.e. cloned) into the checkout_dir, then the constructor will raise an SlcError.

Additionally, the caller needs to ensure, that a flavor with this name is contained in the SLC release specified in variable constants.SLC_RELEASE_TAG.

GITHUB_URL = 'https://github.com/exasol/script-languages-release/releases/tag/10.0.0'

Hyperlink to the GitHub tag which is being used.

property activation_key: str

Returns the language activation string for the uploaded script-language-container. Can be used in ALTER SESSION or ALTER_SYSTEM SQL commands to activate the language of the uploaded script-language-container.

append_custom_conda_packages(conda_packages: list[CondaPackageDefinition])

Appends packages to the custom conda packages file. Note: This method is not idempotent: Multiple calls with the same package definitions will result in duplicated entries.

append_custom_pip_packages(pip_packages: list[PipPackageDefinition])

Appends packages to the custom pip packages file. Note: This method is not idempotent: Multiple calls with the same package definitions will result in duplicated entries.

property checkout_dir: Path

classmethod clean_docker_images(output_path: Path = PosixPath('/tmp/tmpavb0u58l/23cbf1eca7741dfdcd78f9de312267529774cc13')) → None

Deletes all SLC related local docker images. :param output_path: Defines the path where the log files are stored.

classmethod create(secrets: Secrets, name: str, flavor: str, compression_strategy: CompressionStrategy = CompressionStrategy.GZIP) → ScriptLanguageContainer

property custom_conda_file: Path

Returns the path to the custom conda packages file of the flavor

property custom_packages_dir

Returns the path to the custom packages directory of the flavor

property custom_pip_file: Path

Returns the path to the custom pip packages file of the flavor

deploy()

Deploys the current script-languages-container to the database and stores the activation string in the Secure Configuration Storage.

property docker_image_tags: list[str]

Return list of Docker image tags related to the current SLC.

export() → None

Exports the current SLC to the export directory.

export_no_copy() → None

Exports the current SLC to the internal output directory only, without copying to the export directory.

property flavor_path: Path

property language_alias: str

Is case-insensitive.

classmethod list_available_flavors() → list[str]

Functions and Classes in Other Packages

exasol.nb_connector.language_container_activation.get_activation_sql(conf: Secrets) → str

Merges multiple language definitions (i.e. URLs) found in the secret store with language definitions currently registered in the database at the SESSION level. Returns a language activation command which combines all that language definitions.

For details on how an activation command may look like please refer to https://docs.exasol.com/db/latest/database_concepts/udf_scripts/adding_new_packages_script_languages.htm

The secret store entries containing language definitions should have keys with a common prefix. This prefix is defined in the ACTIVATION_KEY_PREFIX.

Multiple language definitions in the secret store would normally have unique aliases. In case there are more than one entry with the same alias, their definitions should match. Otherwise, a RuntimeError will be raised. If a language with the same aliases is already present in the database, its definition will be overwritten, no exception raised.

exasol.nb_connector.language_container_activation.open_pyexasol_connection_with_lang_definitions(conf: Secrets, **kwargs) → ExaConnection

Opens a pyexasol connection and applies the ALTER SESSION command using all registered languages.

class exasol.nb_connector.model_installation.TransformerModel(name: str, task_type: str, factory: Any)

factory: Any

name: str

task_type: str

exasol.nb_connector.model_installation.install_model(conf: Secrets, model: TransformerModel) → None

Download and install the specified Huggingface model.

exasol.nb_connector.sagemaker_extension_wrapper.initialize_sme_extension(conf: Secrets, version: str = '0.11.5', language_alias: str = 'PYTHON3_SME', run_deploy_container: bool = True, run_deploy_scripts: bool = True, run_encapsulate_aws_credentials: bool = True, allow_override: bool = True) → None

Performs all necessary operations to get the Sagemaker Extension up and running. See the “Getting Started” and “Setup” sections of the extension’s User Guide for details.

Parameters:

• conf – The secret store. The store should contain all the required parameters for accessing the database, BucketFS and AWS.

• version – Sagemaker Extension version. If not specified the hardcoded latest known version will be used.

• language_alias – The language alias of the extension’s language container. Normally this parameter would only be used for testing.

• run_deploy_container – If set to False will skip the language container deployment.

• run_deploy_scripts – If set to False will skip the deployment of the scripts.

• run_encapsulate_aws_credentials – If set to False will skip the creation of the database connection object encapsulating the AWS credentials.

• allow_override – If True allows overriding the language definition. Otherwise, if the database already has a language definition for the specified language alias, an attempt to deploy the container will result in a RuntimeError.

exasol.nb_connector.secret_store.Secrets(db_file: Path, master_password: str) → None

class exasol.nb_connector.text_ai_extension_wrapper.Extraction(*, extractor: Annotated[KeywordExtractor | TextSearchExtractor | KeywordSearchExtractor | NamedEntityExtractor | TopicClassifierExtractor | TokenizerExtractor | SourceTableExtractor | StandardExtractor | PipelineExtractor | BranchExtractor | TokenPositionExtractor | CoOccurrenceExtractor | FilterExtractor | SplitterExtractor, AfterValidator(func=validate_source)], output: Output, defaults: Defaults = Defaults(model_repository=None, batch_size=100, parallelism_per_node=4))

defaults_with_model_repository(conf: Secrets) → Defaults

model_config: ClassVar[ConfigDict] = {'frozen': True, 'ignored_types': (<class 'exasol.ai.text.impl.utils.pydantic_cython_helper.CyFunctionDetector'>,)}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

run(conf: Secrets) → None

exasol.nb_connector.text_ai_extension_wrapper.initialize_text_ai_extension(conf: Secrets, container_file: Path | None = None, version: str | None = None, install_slc: bool = True, install_scripts: bool = True, install_models: bool = True, allow_override_language_alias: bool = True) → None

Depending on which flags are set, runs different steps to install Text-AI Extension in the DB. Possible steps:

• Call the Text-AI Extension’s language container deployment API.

If given a version, downloads the specified released version of the extension and uploads it to the BucketFS.

If given a container_file path instead, installs the given container in the Bucketfs.

If neither is given, attempts to install the latest version.

This function doesn’t activate the language container. Instead, it gets the activation SQL using the same API and writes it to the secret store. The name of the key is defined in the ACTIVATION_KEY constant.

• Install default transformers models into the Bucketfs using Transformers Extensions upload model functionality.

• Install Text-AI specific scripts.

Parameters:

• conf – The secret store. The store must contain the DB connection parameters and the parameters of the BucketFS service.

• container_file – Optional. Path pointing to the locally stored Script Language Container file for the Text-AI Extension.

• version – Optional. Text-AI extension version.

• install_slc – If True runs deployment of the locally stored Script Language Container file for the Text-AI Extension.

• install_scripts – If True runs deployment of Text-AI Extension scripts.

• install_models – If True uploads default Transformers models to the BucketFS.

• install_bfs_credentials – If set to False will skip the creation of the text ai specific database connection object encapsulating the BucketFS credentials.

• allow_override_language_alias – If True allows overriding the language definition.

exasol.nb_connector.transformers_extension_wrapper.initialize_te_extension(conf: Secrets, version: str = '2.3.0', language_alias: str = 'PYTHON3_TE', run_deploy_container: bool = True, run_deploy_scripts: bool = True, run_encapsulate_bfs_credentials: bool = True, run_encapsulate_hf_token: bool = True, allow_override: bool = True) → None

Performs all necessary operations to get the Transformers Extension up and running. See the “Getting Started” and “Setup” sections of the extension’s User Guide for details.

Parameters:

• conf – The secret store. The store should contain all the required parameters for accessing the database and BucketFS.

• version – Transformers Extension version. If not specified the hardcoded latest known version will be used.

• language_alias – The language alias of the extension’s language container. Normally this parameter would only be used for testing.

• run_deploy_container – If set to False will skip the language container deployment.

• run_deploy_scripts – If set to False will skip the deployment of the UDF scripts.

• run_encapsulate_bfs_credentials – If set to False will skip the creation of the database connection object encapsulating the BucketFS credentials.

• run_encapsulate_hf_token – If set to False will skip the creation of the database connection object encapsulating the Huggingface token.

• allow_override – If True allows overriding the language definition. Otherwise, if the database already has a language definition for the specified language alias, an attempt to deploy the container will result in a RuntimeError.

exasol.nb_connector.utils.upward_file_search(file_name: str) → str

Looks for a specified file starting from the current directory upward the file hierarchy. Hence, the last place to find the file is the root.

Returns the full path of the file if found. Otherwise, raises a ValueError exception.