Manage data quality rules¶
Experimental feature
Data quality rules are currently an experimental feature. The functionality may change in future releases.
Create data quality rules¶
Data quality rules can be created using three different creator methods depending on the type of rule you want to create:
- Column level rules: For rules that apply to specific columns (e.g., Freshness, Null Count)
- Table level rules: For rules that apply to entire tables (e.g., Row Count)
- Custom SQL rules: For Custom SQL rule only
Column level rules¶
Column level rules are used for data quality checks that apply to specific columns within a asset.
Coming soon
Create a column level data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
|
- Use the
column_level_rule_creator
method to create column-level data quality rules. - Provide the Atlan client instance.
- Specify the rule type (e.g., "Freshness", "Null Count"). The rule type must match exactly what is shown in the UI.
- Reference the asset using its qualified name to which you want to apply this rule.
- Reference the specific column using its qualified name of that asset to which you want to apply this rule. Ensure the column data type is compatible with the rule type (e.g., date/time columns for Freshness rules).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Optional: Specify the threshold unit (e.g., DAYS, HOURS) for rules that support units (such as Freshness). For rules without units (such as Null Count), omit this parameter.
- Save the data quality rule to Atlan.
- Optional: Specify the threshold compare operator (same as you would in the UI).
Coming soon
Coming soon
Table Level rules¶
Table level rules are used for data quality checks that apply to entire table.
Coming soon
Create a Table Level data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
- Use the
table_level_rule_creator
method to create table-level data quality rules. - Provide the Atlan client instance.
- Specify the rule type (e.g., "Row Count"). The rule type must match exactly what is shown in the UI.
- Reference the asset using its qualified name to which you want to apply this rule.
- Set the threshold comparison operator (e.g., EQUAL, LESS_THAN_EQUAL).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Save the data quality rule to Atlan.
Coming soon
Coming soon
Custom SQL rule¶
Custom SQL rule allow you to define data quality check using custom SQL queries.
Coming soon
Create a Custom SQL data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
- Use the
custom_sql_creator
method to create custom SQL data quality rules. - Provide the Atlan client instance.
- Provide a name for the custom rule (same as you would in the UI).
- Reference the asset using its qualified name to which you want to apply this rule.
- Provide the custom SQL query for the rule (same as you would in the UI).
- Set the threshold comparison operator (same as you would in the UI).
- Set the threshold value for the rule (same as you would in the UI).
- Set the alert priority level (same as you would in the UI).
- Set the data quality dimension (e.g., COMPLETENESS, ACCURACY) (same as you would in the UI).
- Optional: Provide a description for the rule.
- Save the data quality rule to Atlan.
Coming soon
Coming soon
Update data quality rules¶
To update an existing data quality rule, you only need to provide the qualified name
and the Atlan Client
. All other parameters are optional and will only be updated if provided.
Coming soon
Update a data quality rule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
|
- Use the
updater
method to update an existing data quality rule. - Provide the Atlan client instance.
- Provide the qualified name of the existing rule.
- Optional: Update the threshold comparison operator.
- Optional: Update the threshold value for the rule.
- Optional: Update the alert priority level.
- Optional: Update the threshold unit.
- Optional: Update the data quality dimension (for custom SQL rules).
- Optional: Update the custom SQL query (for custom SQL rules).
- Optional: Update the name of the rule (for custom SQL rules).
- Optional: Update the description of the rule (for custom SQL rules).
- Save the updated data quality rule to Atlan.
Coming soon
Coming soon
UI compatibility required
When updating data quality rules, only update parameters that are applicable to your specific rule type as shown in the UI. Updating parameters that don't apply to your rule type may cause the operation to fail or produce unexpected results.
Retrieve data quality rules¶
To retrieve data quality rules, you can use fluent search to retireve data quality rules.
Coming soon
Retrieve data quality rules | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
|
- Create a Fluent Search request to retrieve data quality rules from a specific connection.
- Execute the search request to retrieve the data quality rules.
- Iterate through all matching data quality rules and print their details.
Coming soon
Coming soon
Delete data quality rules¶
To delete data quality rules, you can use the standard asset deletion method.
Coming soon
Soft-delete data quality rules | |
---|---|
1 2 3 4 5 6 7 |
|
- Use the
asset.delete_by_guid()
method to delete a data quality rule. Provide the GUID of the rule you want to delete. - The
assets_deleted(asset_type=alpha_DQRule)
method returns a list of the assets of the given type that were deleted. - If an asset of the given type was deleted, then the deleted form of the asset is available.
Coming soon
Coming soon
Schedule data quality rules¶
To add a schedule for data quality rules on an asset, you can use the add_dq_rule_schedule
method. This method allows you to set up the schedule for data quality rule execution.
Coming soon
Add data quality rule schedule | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 |
|
- Use the
add_dq_rule_schedule
method to add a schedule for data quality rules on an asset. - Specify the asset type (e.g., Table).
- Provide the name of the asset as it appears in Atlan.
- Provide the qualified name of the asset (same as you would see in the Atlan UI).
- Provide the cron schedule string following the standard cron format (e.g., "41 20 * 1 *" means run at 20:41 only in January).
- Provide the timezone string in the format used by Atlan UI (e.g., "Europe/Paris", "Asia/Calcutta").
Coming soon
POST /api/meta/entity/bulk | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
- The asset GUID to which the DQ rule schedule needs to be implemented.
- The type of the asset (e.g., "Table").
- The name of the asset.
- The qualified name of the asset.
- Set to "CRON" for cron-based scheduling.
- The cron schedule string (e.g., "41 20 * * 0,1,4-6").
- The timezone string (e.g., "Asia/Calcutta").
Cron Schedule Format
The standard cron schedule format consists of five fields, separated by spaces:
- Minute (0-59): The minute of the hour when the command will run
- Hour (0-23): The hour of the day when the command will run (0 is midnight, 23 is 11 PM)
- Day of Month (1-31): The day of the month when the command will run
- Month (1-12): The month of the year when the command will run (1 is January, 12 is December)
- Day of Week (0-6): The day of the week when the command will run (0 is Sunday, 1 is Monday, and so on up to 6 for Saturday)