deirokay.statements.builtin.column_expression.ColumnExpression

class deirokay.statements.builtin.column_expression.ColumnExpression(*args, **kwargs)[source]

Bases: BaseStatement

Evaluates an expression (or a list of expressions) involving the scope columns, using numpy.eval(). The statement passes only if all expressions evaluate to True.

The columns in the scope must be ideally of the same dtype. This statement supports the following dtypes: string, integer, float and decimal.

The available parameters for this statement are:

expressions (required): an expression (or a list of expressions) to be evaluated. The valid operators are: ==, !=, =~, >=, <=, > and <.
at_least_%: the minimum percentage of valid rows. Default: 100.
at_most_%: the maximum percentage of valid rows. Default: 100.
rtol: the relative tolerance for float evaluations (when using the =~ operator). Default: 1e-5.
atol: the absolute tolerance for float evaluations (when using the =~ operator). Default: 1e-8.

Examples

In the example below, in JSON format, we test whether or not the values of the a column are equal to the values of the b column. Similarly, we test whether or not the values of the b column are greater than the values of the c column:

{
    "scope": ["a", "b", "c"],
    "statements": [
        {
            "type": "column_expression",
            "expressions": ["a == b", "a < c"],
            "at_least_%": 50.0
        }
    ]
}

For float comparisons, you may prefer using the rtol or atol parameters, in addition to the =~ operator. For example, if you want to test whether or not the values of the a column are equal to the values of the b column with a relative tolerance of 1e-3, you can use the following JSON:

{
    "scope": ["a", "b"],
    "statements": [
        {
            "type": "column_expression",
            "expressions": "a =~ b",
            "rtol": 1e-3
        }
    ]
}

Methods

`attach_backend`	Generate a subclass that concretizes multibackend backend methods into their intended name.
`get_backend`	Get current active backend for this class.
`profile`	Given a template data table, generate a statement dict from it.
`register_backend_method`	Proxy for register_backend_method to register an existing function as a backend-specific method.
`report`	Receive a DataFrame containing only columns on the scope of validation and returns a report of related metrics that can be used later to declare this Statement as fulfilled or failed.
`result`	Receive the report previously generated and declare this statement as either fulfilled (True) or failed (False).

Attributes

`VALID_OPERATORS`
`expected_parameters`	Parameters expected for this statement.
`name`	Statement name when referred in Validation Documents (only valid for Deirokay built-in statements).
`supported_backends`	Backends supported by this resource.

VALID_OPERATORS = '==|!=|=~|>=|<=|>|<'

__call__(df: DeirokayDataSource) → dict: Run statement instance.

classmethod __init_subclass__() → None: Validate subclassed statement.

classmethod __post_attach_backend__(): This classmethod can be optionally overwritten to serve as a callback function for when the attach_backend() method is called.

classmethod attach_backend(backend: Backend) → Type[_AnyMultiBackendClass]

Generate a subclass that concretizes multibackend backend methods into their intended name. The methods marked with the given backend will compose the returned class.

Parameters

cls (type) – Class to be subclassed with the given backend.
backend (Backend) – Backend to be selected.

Returns

Subclass of the current class with methods filtered for the given backend.

Return type

Type[MultiBackendMixin]

expected_parameters: List[str] = ['expressions', 'at_least_%', 'at_most_%', 'rtol', 'atol']

Parameters expected for this statement.

Type: List[str]

classmethod get_backend() → Backend

Get current active backend for this class.

Returns: The current active backend.
Return type: Backend
Raises: InvalidBackend – Backend not set or not a valid execution class.

name: str = 'column_expression'

Statement name when referred in Validation Documents (only valid for Deirokay built-in statements).

Type: str

static profile(df: DeirokayDataSource) → Dict[str, Any]

Given a template data table, generate a statement dict from it.

Parameters: df (DataFrame) – The DataFrame to be used as template.
Returns: Statement dict.
Return type: dict
Raises: NotImplementedError – If this method is not implemented by the subclass or the profile generation for this statement was intentionally skipped.

classmethod register_backend_method(alias_for: str, func: Callable[[...], Any], backend: Backend) → None

Proxy for register_backend_method to register an existing function as a backend-specific method.

Parameters

alias_for (str) – The name of the method to be substituted with a backend-specific version.
func (AnyCallable) – Existing function to be registered as a method.
backend (Backend) – Backend for the method.

report(df: DeirokayDataSource) → dict

Receive a DataFrame containing only columns on the scope of validation and returns a report of related metrics that can be used later to declare this Statement as fulfilled or failed.

Parameters: df (DataFrame) – The scoped DataFrame columns to be analysed in this report by this statement.
Returns: A dictionary of useful statistics about the target columns.
Return type: dict

result(report: dict) → bool[source]

Receive the report previously generated and declare this statement as either fulfilled (True) or failed (False).

Parameters: report (dict) – Report generated by report method. Should ideally contain all statistics necessary to evaluate the statement validity.
Returns: Whether or not this statement passed.
Return type: bool

supported_backends: List[Backend] = [<Backend.PANDAS: 'pandas'>, <Backend.DASK: 'dask'>]

Backends supported by this resource.

Type: List[Backend]