deirokay.statements.builtin.not_null.NotNull

class deirokay.statements.builtin.not_null.NotNull(*args, **kwargs)[source]

Bases: BaseStatement

Check if the rows of a scoped DataFrame are not null, possibly setting boundaries for the minimum and maximum percentage of not-null rows.

The available options are:

at_least_%: The minimum percentage of not-null rows. Default: 100.0.
at_most_%: The maximum percentage of not-null rows. Default: 100.0.
multicolumn_logic: The logic to use when checking for not-null values in multicolumn scopes (either ‘any’ or ‘all’). Default: ‘any’.

Be careful When using multicolumn scopes: the any logic considers a row as null only if all columns are null. The all logic considers a row as null when any of its columns is null.

Examples

You want to ensure that less than 1% of the values in a column foo are null. You can declare the following validation item:

{
    "scope": "foo",
    "statements": [
        {
            "name": "not_null",
            "at_least_%": 99.0
        }
    ]
}

You noticed that you imposed a unrealistic value for at_least_%, and maybe less than 10% should be a reasonable percentage of null values. Still, you don’t want to lose track of that ideal <= 1% checks, since you intend to improve your data quality in the near future. You may take advantage of severity to set different exception levels for different values of at_least_%:

{
    "scope": "foo",
    "statements": [
        {
            "name": "not_null",
            "at_least_%": 99.0,
            "severity": 3
        },
        {
            "name": "not_null",
            "at_least_%": 90.0,
            "severity": 5
        }
    ]
}

This way, values between 90% and 99% will only raise a warning, while values below 90% will raise a validation exception (by default).

You don’t tolerate any null values in a list of columns:

{
    "scope": ["foo", "bar", "baz", "qux"],
    "statements": [
        {
            "name": "not_null",
            "multicolumn_logic": "all"
        }
    ]
}

Methods

`attach_backend`	Generate a subclass that concretizes multibackend backend methods into their intended name.
`get_backend`	Get current active backend for this class.
`profile`	Given a template data table, generate a statement dict from it.
`register_backend_method`	Proxy for register_backend_method to register an existing function as a backend-specific method.
`report`	Receive a DataFrame containing only columns on the scope of validation and returns a report of related metrics that can be used later to declare this Statement as fulfilled or failed.
`result`	Receive the report previously generated and declare this statement as either fulfilled (True) or failed (False).

Attributes

`expected_parameters`	Parameters expected for this statement.
`name`	Statement name when referred in Validation Documents (only valid for Deirokay built-in statements).
`supported_backends`	Backends supported by this resource.

__call__(df: DeirokayDataSource) → dict: Run statement instance.

classmethod __init_subclass__() → None: Validate subclassed statement.

classmethod __post_attach_backend__(): This classmethod can be optionally overwritten to serve as a callback function for when the attach_backend() method is called.

classmethod attach_backend(backend: Backend) → Type[_AnyMultiBackendClass]

Generate a subclass that concretizes multibackend backend methods into their intended name. The methods marked with the given backend will compose the returned class.

Parameters

cls (type) – Class to be subclassed with the given backend.
backend (Backend) – Backend to be selected.

Returns

Subclass of the current class with methods filtered for the given backend.

Return type

Type[MultiBackendMixin]

expected_parameters: List[str] = ['at_least_%', 'at_most_%', 'multicolumn_logic']

Parameters expected for this statement.

Type: List[str]

classmethod get_backend() → Backend

Get current active backend for this class.

Returns: The current active backend.
Return type: Backend
Raises: InvalidBackend – Backend not set or not a valid execution class.

name: str = 'not_null'

Statement name when referred in Validation Documents (only valid for Deirokay built-in statements).

Type: str

static profile(df: DeirokayDataSource) → Dict[str, Any]

Given a template data table, generate a statement dict from it.

Parameters: df (DataFrame) – The DataFrame to be used as template.
Returns: Statement dict.
Return type: dict
Raises: NotImplementedError – If this method is not implemented by the subclass or the profile generation for this statement was intentionally skipped.

classmethod register_backend_method(alias_for: str, func: Callable[[...], Any], backend: Backend) → None

Proxy for register_backend_method to register an existing function as a backend-specific method.

Parameters

alias_for (str) – The name of the method to be substituted with a backend-specific version.
func (AnyCallable) – Existing function to be registered as a method.
backend (Backend) – Backend for the method.

report(df: DeirokayDataSource) → dict

Receive a DataFrame containing only columns on the scope of validation and returns a report of related metrics that can be used later to declare this Statement as fulfilled or failed.

Parameters: df (DataFrame) – The scoped DataFrame columns to be analysed in this report by this statement.
Returns: A dictionary of useful statistics about the target columns.
Return type: dict

result(report: dict) → bool[source]

Receive the report previously generated and declare this statement as either fulfilled (True) or failed (False).

Parameters: report (dict) – Report generated by report method. Should ideally contain all statistics necessary to evaluate the statement validity.
Returns: Whether or not this statement passed.
Return type: bool

supported_backends: List[Backend] = [<Backend.PANDAS: 'pandas'>, <Backend.DASK: 'dask'>]

Backends supported by this resource.

Type: List[Backend]