Define package via template¶
How to read this guide
Each section of this guide provides 2 tabs, which are linked throughout (once you swap in one section, all other sections will automatically reflect that same level of detail):
- Simple — when you are just starting out, follow these tabs to understand the basic structure of the toolkit and the fundamental elements that you must use.
- Detailed — as you start to wonder about additional complexity, consider changing to these tabs, which cover additional (optional) possibilities.
Start by creating a Pkl file that amends our published package config toolkit model:
MyCustomPackage.pkl | |
---|---|
1 |
|
If this is the first time you're creating a package, hover over that line and download the Pkl package.
Define overall metadata¶
Then you can start defining your package. All packages should have at least these main components:
MyCustomPackage.pkl | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
|
- Import the standard Pkl library for semantic versioning, to define your package's version.
- (Optional) If your package could create a connection, import the pre-defined set of connections from the package toolkit.
- A
packageId
, which gives a unique namespaced identity for your package. It must be of the form@namespace/kebab-case-name
. - A
packageName
, which gives the human-readable name of your package, as it should show in the UI. - A
version
for your package, which must follow semantic versioning. - A
description
for your package, as it should show in the UI. - An
iconUrl
giving an online location where the icon to use in the UI for this package can be accessed. - A
docsUrl
giving an online location where additional documentation about the package can be found. This will be linked in the UI. - The
implementationLanguage
defining the coding language you will use to implement the logic of your package. This is used by the toolkit to generate a strongly-typed data class that captures the inputs a user provides during setup of the package. -
The
containerImage
that will encapsulate your package's code.Use variables where possible
Note that Pkl is itself a language, so you can use variables here as well (for example, if your container images follow a naming convention that can be generated from the
packageId
or to automatically set the version). -
The full
containerCommand
used to run your custom logic within the container. This should be specified as a list, rather than a single spaced-out string. - (Optional) Any
outputs
your package's logic will produce, that you want a user to be able to download. In this example, our package will write debug logs out to/tmp/debug.log
and we will make these available for download. - (Optional) Any
keywords
you want your package to have associated with it. - (Optional) The
category
controls the name of the pill in the marketplace in the UI where the package will be listed. It will default tocustom
if not specified. - (Optional) Setting
preview
to true will show a small orange badge on the package indicating it is experimental (by default it will not be experimental). - (Optional) Setting
certified
to false will remove the green tick badge from the package indicating it is certified (by default it will be certified). - (Optional) When your package could create a connection, use the
connectorType
to define what kind of connection it should create. - A
uiConfig
section where you will define the inputs you want your package to receive from the user during setup. - (Optional) A
credentialConfig
section where you can define any sensitive inputs you want the user to provide, typically to connect to an external system. Any details provided by a user within these inputs will be encrypted and stored in a secure vault in Atlan.
Define configurable inputs¶
Use the uiConfig
section to define the configurable inputs you want the user to provide when setting up your package.
These inputs fit into a two-level hierarchy:
- Tasks are the top-level steps you see along the left side when setting up a new package.
- Inputs are the individual widgets a user can enter information into, select from, etc within each task (step).
In addition to these inputs, you can (optionally) define one or more rules to control which inputs appear on the screen, based on what a user has selected in some other input.
MyCustomPackage.pkl | |
---|---|
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
|
- You must define one (and only one) list of
tasks
. - Use the name as you want it to show in the UI for each top-level step, surrounded in square brackets. You can then enclose the definition of that top-level step in curly braces.
- Include a description, which will show in the UI as part of the top-level step (very space-limited).
- Also include the list of
inputs
that should be configured as part of that top-level step. - You can define as many of these top-level steps as you like. They will be displayed in the order (top-down) they are listed here in the model.
- (Optional) You can then specify any rules to control which inputs appear on the screen (or not), based on what a user has selected in some other input.
MyCustomPackage.pkl | |
---|---|
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
|
- You must define one (and only one) list of
tasks
. - Use the name as you want it to show in the UI for each top-level step, surrounded in square brackets. You can then enclose the definition of that top-level step in curly braces.
- Include a description, which will show in the UI as part of the top-level step (very space-limited).
- Also include the list of
inputs
that should be configured as part of that top-level step. - For each input, specify a unique variable name (in
lower_snake_case
) surrounded by square brackets. You can choose from a variety of widgets to use for receiving that input from a user. - All inputs will at least need a
title
, which defines the text to show in the UI for that widget. - All inputs can also be configured as mandatory using
required = true
(by default they'll be optional). - All inputs can also be given a
helpText
, which when provided shows an information icon next to thetitle
that can be hovered over for more information on how the input should be used. - Other configurable options will vary by the kind of input, but could include for example some light grey text to show in the box by default to give the user some idea of the value you're expecting from them.
- You can define as many of these top-level steps as you like, and within each as many inputs as you like. They will be displayed in the order (top-down) they are listed here in the model.
- (Optional) You can specify a
fallback
value to use for any input, in case the user does not make any selection or enter any value in the UI. - (Optional) You can then specify any rules to control which inputs appear on the screen (or not), based on what a user has selected in some other input.
- Each new rule needs to be defined as
new UIRule
, with its definition enclosed in curly braces. - You must specify a
whenInputs
that defines an input variable name and value. - When the input variable has the value indicated above, all input variables defined in this
required
will be shown in the UI (otherwise they will be hidden). - When you have multiple conditions or inputs to show or hide, you can use this more spaced out form. This example means that when a user has selected
CREATE
for theconnection_usage
input, the UI will show theconnection
input.
Define sensitive inputs¶
Use the credentialConfig
section to define sensitive inputs you want the user to provide when setting up your package.
Encrypted and secured in a vault
Any details a user provides within this section will be encrypted and secured in a vault within Atlan, for added protection.
These inputs fit into two types:
- Common inputs are inputs that are common irrespective of the type of credential the user is configuring.
- Options define the different types of credentials a user can configure (for example, basic username and password vs shared secrets).
MyCustomPackage.pkl | |
---|---|
39 40 41 42 43 44 45 46 47 48 49 50 51 |
|
- You must give the credentials configuration a unique name. (This allows you to reuse credential configurations without needing to redefine them in different packages.)
- Set the
source
to the type of connector through which these credentials can be used. - Define the icon to use when representing this particular source and credential in the UI.
- Provide a link to any documentation about how to set up these credentials, for example any pre-requites on the permissions that are required on the external system.
- Define the logo to use when representing this particular source and credential in the UI.
- (Optional) You can define any common inputs that are shared across different connectivity options.
- You must define at least one option for connecting to the external system, and could define many. For example, you may want to support both a username and password as one option, and another option that uses some shared secrets or API tokens.
MyCustomPackage.pkl | |
---|---|
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
|
- You must give the credentials configuration a unique name. (This allows you to reuse credential configurations without needing to redefine them in different packages.)
- Set the
source
to the type of connector through which these credentials can be used. - Define the icon to use when representing this particular source and credential in the UI.
- Provide a link to any documentation about how to set up these credentials, for example any pre-requites on the permissions that are required on the external system.
- Define the logo to use when representing this particular source and credential in the UI.
- (Optional) You can define any common inputs that are shared across different connectivity options.
- For each input, specify a unique variable name (in
lower_snake_case
) surrounded by square brackets. You can choose from a variety of widgets to use for receiving that input from a user. - All inputs will need at least a
title
, which defines the text to show in the UI for that widget. - Other configurable options will vary by the kind of input, but could include for example some light grey text to show in the box by default to give the user some idea of the value you're expecting from them.
- You must define at least one option for connecting to the external system, and could define many. For example, you may want to support both a username and password as one option, and another option that uses some shared secrets or API tokens.
Delegate publishing¶
You can now use the toolkit to delegate publishing of assets through packages that use CSV inputs. Your own logic simply needs to produce a CSV output file that matches the required format of one of these publishing package's CSV inputs.
Handles many common scenarios automatically
These publishing packages are designed to handle common scenarios for you directly, such as:
- Calculating differences (deltas) between a previous load and the current load, to automatically:
- Identify which assets should be deleted (or archived), and do so
- Optimize the publishing to only save any assets that have changed
- Support only updating assets (avoiding any accidental asset creation), if desired
- Allow case-insensitive matching of
qualifiedName
s of assets - Resolve any ambiguity between tables or views for you, automatically
To add a delegated publish step to your package, you need only add a publishConfig
to your template:
MyCustomPackage.pkl | |
---|---|
115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
|
- When configuring which publish package to delegate, you can choose from either
AssetImport
,RelationalAssetsBuilder
orLineageBuilder
. Each can be configured with different options, which are described further in the sections below. - You need to provide the specific version of the package you want to use for publishing. This version will match the version published at atlanhq/atlan-java.
-
For any file-based inputs, use the toolkit's
transferFile()
helper to indicate how to transfer the file from the outputs your own package produces:outputs
is the name of the variable in your package defining its outputs (will always be this)"transformed_file"
is the key of the entry in thatoutputs
variable that points to the file your package produces that you want to delegate for publishing"current.csv"
is the name of the file you want to use when you pass it along to the publishing package
-
For any configuration-based inputs, use the toolkit's
transferConfigInput()
helper to indicate which UI-based input variable's value to transfer to the publishing package.uiConfig
is the name of the variable in your package defining its UI-based inputs (will always be this)"assets_upsert_semantic"
is the name of the UI input (widget variable) from which to take the value to be transferred
-
You can also use a literal primitive value for any of the options, if you just want to hard-code it rather than transfer anything across from the UI-based configuration or your own package's outputs.
AssetImport¶
The AssetImport
publish config accepts the following configurable inputs. You can use any combination of these, and any you leave out will use sensible defaults.
Glossaries, categories, terms
Variable Name | Data Type | Description |
---|---|---|
glossariesFile | FrameworkRenderer.NamePathPair | File containing glossaries, categories and terms to import; typically passed through using transferFile() . |
glossariesUpsertSemantic | ImportSemantic | Whether to allow the creation of new glossaries, categories and terms from the input CSV, or ensure these are only updated if they already exist in Atlan. |
glossariesConfig | ConfigType | Options to optimize how glossaries, categories and terms are imported. |
glossariesAttrToOverwrite | Listing<String> | List of attributes you want to clear (remove) from glossaries, categories and terms if their value is blank in the provided file. |
glossariesFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
glossariesFieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
glossariesCmHandling | CustomMetadataSemantic | How custom metadata in the input should be handled: ignore it, merge it with any existing asset custom metadata, or overwrite the existing asset custom metadata. |
glossariesTagHandling | AtlanTagSemantic | How Atlan tags on assets in the input should be handled: ignore them, append them to any existing asset tags, replace the existing asset tags, or remove them from the assets. |
glossariesLinkIdempotency | LinkIdempotencyInvariant | How linked resources on glossaries in the input should be updated: based on their unique URL or their unique name. |
glossariesBatchSize | Int | Maximum number of rows to process at a time (per API request). |
Data domains and products
Variable Name | Data Type | Description |
---|---|---|
dataProductsFile | FrameworkRenderer.NamePathPair | File containing data domains and data products to import; typically passed through using [transferFile()]. |
dataProductsUpsertSemantic | ImportSemantic | Whether to allow the creation of new domains and data products from the input CSV, or ensure these are only updated if they already exist in Atlan. |
dataProductsConfig | ConfigType | Options to optimize how domains and data products are imported. |
dataProductsAttrToOverwrite | Listing<String> | Select attributes you want to clear (remove) from domains and data products if their value is blank in the provided file. |
dataProductsFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
dataProductsFieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
dataProductsCmHandling | CustomMetadataSemantic | How custom metadata in the input should be handled: ignore it, merge it with any existing asset custom metadata, or overwrite the existing asset custom metadata. |
dataProductsTagHandling | AtlanTagSemantic | How Atlan tags on assets in the input should be handled: ignore them, append them to any existing asset tags, replace the existing asset tags, or remove them from the assets. |
dataProductsLinkIdempotency | LinkIdempotencyInvariant | How linked resources on assets in the input should be updated: based on their unique URL or their unique name. |
dataProductsBatchSize | Int | Maximum number of rows to process at a time (per API request). |
Tag (structural) definitions
Variable Name | Data Type | Description |
---|---|---|
tagsFile | FrameworkRenderer.NamePathPair | File containing tag definitions to manage; typically passed through using transferFile() . |
tagsConfig | ConfigType | Options to optimize how tag definitions are imported. |
tagsFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
tagsFieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
tagsBatchSize | Int | Maximum number of rows to process at a time (per API request). |
All other assets
Variable Name | Data Type | Description |
---|---|---|
assetsFile | FrameworkRenderer.NamePathPair | File containing assets to import, typically passed through using transferFile() . |
assetsUpsertSemantic | ImportSemantic | Whether to allow the creation of new assets from the input CSV (full or partial assets), or ensure assets are only updated if they already exist in Atlan. |
assetsDeltaSemantic | DeltaSemantic | Whether to treat the input file as an initial load, full replacement (deleting any existing assets not in the file) or only incremental (no deletion of existing assets). |
assetsDeltaRemovalType | RemovalType | How to delete any assets not found in the latest file. |
assetsDeltaReloadCalculation | ReloadCalculation | Which assets to reload from the latest input CSV file. Changed assets only will calculate which assets have changed between the files and only attempt to reload those changes. |
assetsPreviousFileDirect | FrameworkRenderer.NamePathPair | Path to a direct file (locally) to use for delta processing. Note: providing a value for this will ignore any other previously-processed file in the object store, so please be sure this is the option you want to use (should be rare). |
assetsPreviousFilePrefix | String | Object store prefix in which previous files exist for delta processing. |
assetsConfig | ConfigType | Options to optimize how assets are imported. |
assetsAttrToOverwrite | Listing<String> | List of attributes you want to clear (remove) from assets if their value is blank in the provided file. |
assetsFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
assetsCaseSensitive | Boolean | Whether to use case-sensitive matching when running in update-only mode (true) or try case-insensitive matching (false). |
assetsTableViewAgnostic | Boolean | Whether to treat tables, views and materialized views as interchangeable (true) or strictly adhere to specified types in the input (false). |
assetsFieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
assetsCmHandling | CustomMetadataSemantic | How custom metadata in the input should be handled: ignore it, merge it with any existing asset custom metadata, or overwrite the existing asset custom metadata. |
assetsTagHandling | AtlanTagSemantic | How Atlan tags on assets in the input should be handled: ignore them, append them to any existing asset tags, replace the existing asset tags, or remove them from the assets. |
assetsLinkIdempotency | LinkIdempotencyInvariant | How linked resources on assets in the input should be updated: based on their unique URL or their unique name. |
assetsBatchSize | Int | Maximum number of rows to process at a time (per API request). |
trackBatches | Boolean | Whether to track details about every asset across batches (true) or only counts (false). |
RelationalAssetsBuilder¶
The RelationalAssetsBuilder
publish config accepts the following configurable inputs. You can use any combination of these, and any you leave out will use sensible defaults.
Relational assets
Variable Name | Data Type | Description |
---|---|---|
assetsFile | FrameworkRenderer.NamePathPair | File containing assets to import, typically passed through using transferFile() . |
assetsUpsertSemantic | ImportSemantic | Whether to allow the creation of new (full or partial) assets from the input CSV, or ensure assets are only updated if they already exist in Atlan. |
deltaSemantic | DeltaSemantic | Whether to treat the input file as an initial load, full replacement (deleting any existing assets not in the file) or only incremental (no deletion of existing assets). |
deltaRemovalType | RemovalType | How to delete any assets not found in the latest file. |
deltaReloadCalculation | ReloadCalculation | Which assets to reload from the latest input CSV file. Changed assets only will calculate which assets have changed between the files and only attempt to reload those changes. |
previousFileDirect | FrameworkRenderer.NamePathPair | Path to a direct file (locally) to use for delta processing. Note: providing a value for this will ignore any other previously-processed file in the object store, so please be sure this is the option you want to use (should be rare). |
assetsAttrToOverwrite | Listing<String> | List of attributes you want to clear (remove) from assets if their value is blank in the provided file. |
assetsFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
assetsFieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
assetsBatchSize | Int | Maximum number of rows to process at a time (per API request). |
assetsCmHandling | CustomMetadataSemantic | How custom metadata in the input should be handled: ignore it, merge it with any existing asset custom metadata, or overwrite the existing asset custom metadata. |
assetsTagHandling | AtlanTagSemantic | How Atlan tags on assets in the input should be handled: ignore them, append them to any existing asset tags, replace the existing asset tags, or remove them from the assets. |
trackBatches | Boolean | Whether to track details about every asset across batches (true) or only counts (false). |
LineageBuilder¶
The LineageBuilder
publish config accepts the following configurable inputs. You can use any combination of these, and any you leave out will use sensible defaults.
Lineage assets
Variable Name | Data Type | Description |
---|---|---|
lineageFile | FrameworkRenderer.NamePathPair | File containing lineage to import, typically passed through using transferFile() . |
lineageUpsertSemantic | ImportSemantic | Whether to allow the creation of new (full or partial) assets from the input CSV, or ensure assets are only updated if they already exist in Atlan. |
lineageFailOnErrors | Boolean | Whether an invalid value in a field should cause the import to fail (true) or log a warning, skip that value, and proceed (false). |
fieldSeparator | String | Single character used to separate fields in the input file (for example, , or ; ). |
batchSize | Int | Maximum number of rows to process at a time (per API request). |
cmHandling | CustomMetadataSemantic | How custom metadata in the input should be handled: ignore it, merge it with any existing asset custom metadata, or overwrite the existing asset custom metadata. |
tagHandling | AtlanTagSemantic | How Atlan tags on assets in the input should be handled: ignore them, append them to any existing asset tags, replace the existing asset tags, or remove them from the assets. |