Manage data quality rules¶
Experimental feature
Data quality rules are currently an experimental feature. The functionality may change in future releases.
Create data quality rules¶
Data quality rules can be created using three different creator methods depending on the type of rule you want to create:
- Column level rules: For rules that apply to specific columns (e.g., Freshness, Null Count)
- Table level rules: For rules that apply to entire tables (e.g., Row Count)
- Custom SQL rules: For Custom SQL rule only
Column level rules¶
Column level rules are used for data quality checks that apply to specific columns within a asset.
Coming soon
Create a column level data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
|
- Use the
column_level_rule_creator
method to create column-level data quality rules. - Provide the Atlan client instance.
- Specify the rule type (e.g., "Freshness", "Null Count"). The rule type must match exactly what is shown in the UI.
- Reference the asset using its qualified name to which you want to apply this rule.
- Reference the specific column using its qualified name of that asset to which you want to apply this rule. Ensure the column data type is compatible with the rule type (e.g., date/time columns for Freshness rules).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Optional: Specify the threshold unit (e.g., DAYS, HOURS) for rules that support units (such as Freshness). For rules without units (such as Null Count), omit this parameter.
- Save the data quality rule to Atlan.
- Optional: Specify the threshold compare operator (same as you would in the UI).
Coming soon
Coming soon
Table Level rules¶
Table level rules are used for data quality checks that apply to entire table.
Coming soon
Create a Table Level data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
- Use the
table_level_rule_creator
method to create table-level data quality rules. - Provide the Atlan client instance.
- Specify the rule type (e.g., "Row Count"). The rule type must match exactly what is shown in the UI.
- Reference the asset using its qualified name to which you want to apply this rule.
- Set the threshold comparison operator (e.g., EQUAL, LESS_THAN_EQUAL).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Save the data quality rule to Atlan.
Coming soon
Coming soon
Custom SQL rule¶
Custom SQL rule allow you to define data quality check using custom SQL queries.
Coming soon
Create a Custom SQL data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
- Use the
custom_sql_creator
method to create custom SQL data quality rules. - Provide the Atlan client instance.
- Provide a name for the custom rule (same as you would in the UI).
- Reference the asset using its qualified name to which you want to apply this rule.
- Provide the custom SQL query for the rule (same as you would in the UI).
- Set the threshold comparison operator (same as you would in the UI).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Set the data quality dimension (e.g., COMPLETENESS, ACCURACY) (same as you would in the UI).
- Optional: Provide a description for the rule.
- Save the data quality rule to Atlan.
Coming soon
Coming soon
Update data quality rules¶
To update an existing data quality rule, you only need to provide the qualified name
and the Atlan Client
. All other parameters are optional and will only be updated if provided.
Coming soon
Update a data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
|
- Use the
updater
method to update an existing data quality rule. - Provide the Atlan client instance.
- Provide the qualified name of the existing rule.
- Optional: Update the threshold comparison operator.
- Optional: Update the threshold value for the rule.
- Optional: Update the alert priority level.
- Optional: Update the threshold unit.
- Optional: Update the data quality dimension (for custom SQL rules).
- Optional: Update the custom SQL query (for custom SQL rules).
- Optional: Update the name of the rule (for custom SQL rules).
- Optional: Update the description of the rule (for custom SQL rules).
- Save the updated data quality rule to Atlan.
Coming soon
Coming soon
UI compatibility required
When updating data quality rules, only update parameters that are applicable to your specific rule type as shown in the UI. Updating parameters that don't apply to your rule type may cause the operation to fail or produce unexpected results.
Retrieve data quality rules¶
To retrieve data quality rules, you can use fluent search to retireve data quality rules.
Coming soon
Retrieve data quality rules | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
|
- Create a Fluent Search request to retrieve data quality rules from a specific connection.
- Execute the search request to retrieve the data quality rules.
- Iterate through all matching data quality rules and print their details.
Coming soon
Coming soon
Delete data quality rules¶
To delete data quality rules, you can use the standard asset deletion method.
Coming soon
Soft-delete data quality rules | |
---|---|
1 2 3 4 5 6 7 |
|
- Use the
asset.delete_by_guid()
method to delete a data quality rule. Provide the GUID of the rule you want to delete. - The
assets_deleted(asset_type=alpha_DQRule)
method returns a list of the assets of the given type that were deleted. - If an asset of the given type was deleted, then the deleted form of the asset is available.
Coming soon
Coming soon