Define typedefs via template¶
How to read this guide
Each section of this guide provides 3 tabs, which are linked throughout (once you swap in one section, all other sections will automatically reflect that same level of detail):
- Simple — when you are just starting out, follow these tabs to understand the basic structure of the toolkit and the fundamental elements that you must use.
- Detailed — as you start to wonder about additional complexity, consider changing to these tabs, which cover additional (optional) possibilities.
- With UX — when you are ready to start experimenting with the user interface for your typedefs, use this tab to provide inputs for generating a baseline set of UX code.
Running example (expand for details)
Throughout the guide, anywhere we are creating portions of the running example you will find a similar expandable section to this one, which explains in more detail what the specific section is adding.
Start by creating a Pkl file that amends our published typedef toolkit model:
MyCustomModel.pkl | |
---|---|
1 |
|
If this is the first time you're creating a model, hover over that line and download the package.
Set the overall structure¶
Then you can start defining your model. All models must have at least two components:
- A
namespace
, which uniquely prefixes all types and attributes in your model (to avoid any collisions with others). - A collection of
customAssetTypes
that define the objects in your model.
If you use only these two components, various other defaults will be generated for you automatically (such as the abstract supertype for your model). You can also override or extend aspects of these generated objects, if you look at the Detailed tab. (And finally, the With UX tab shows further options for configuring the user interface that will be coupled to your model.)
MyCustomModel.pkl | |
---|---|
1 2 3 4 5 6 7 |
|
- The namespace is used for every type in the model (PascalCase). It will also automatically be decapitalized for use as an attribute prefix.
customAssetTypes
describe the objects you want to instantiate in your model.
MyCustomModel.pkl | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
- The namespace is used for every type in the model (PascalCase). It will also automatically be decapitalized for use as an attribute prefix.
- (Optional) You can override the attribute prefix, if you do not simply want to decapitalize the namespace. This prefix will be used for every attribute in the model (camelCase). Pkl provides methods like
decapitalize()
to lowercase only the first letter of a string, ortoLowerCase()
to convert an entire string to lowercase. Or you can of course use a literal string here. - (Optional)
customEnumTypes
describe any lists of valid values (enumerations) you want to be able to use anywhere in your model. - (Optional)
customStructTypes
describe any complex (nested) attributes you want to be able to use anywhere in your model. - (Optional)
supertypeDefinition
configures the abstract supertype for all other asset types you want to be able to instantiate. For example, you would use this section to define attributes that should exist across all objects in your model, or if you want your abstract supertype to extend something other than the default supertype (Catalog
). customAssetTypes
describe the objects you want to instantiate in your model.- (Optional)
customRelationshipTypes
describe the relationships between objects in your model.
MyCustomModel.pkl | |
---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
|
- (Optional) If you intend to include any front-end elements in your definition (such as icons), you will need to import the
Frontend.pkl
portion of the toolkit. - The namespace is used for every type in the model (PascalCase). It will also automatically be decapitalized for use as an attribute prefix.
- (Optional) You can override the attribute prefix, if you do not simply want to decapitalize the namespace. This prefix will be used for every attribute in the model (camelCase). Pkl provides methods like
decapitalize()
to lowercase only the first letter of a string, ortoLowerCase()
to convert an entire string to lowercase. Or you can of course use a literal string here. - (Optional)
customEnumTypes
describe any lists of valid values (enumerations) you want to be able to use anywhere in your model. - (Optional)
customStructTypes
describe any complex (nested) attributes you want to be able to use anywhere in your model. - (Optional)
supertypeDefinition
configures the abstract supertype for all other asset types you want to be able to instantiate. For example, you would use this section to define attributes that should exist across all objects in your model, or if you want your abstract supertype to extend something other than the default supertype (Catalog
). customAssetTypes
describe the objects you want to instantiate in your model.- (Optional)
customRelationshipTypes
describe the relationships between objects in your model. - (Optional)
ui
describes overall user interface setup, such as the filters for the discovery page or the breadcrumb trails to use to show an asset's containment hierarchy.
Define reusable structures¶
(Optional) Use the customEnumTypes
and customStructTypes
sections to define any reusable structures for your model.
Running example (expand for details)
From the running example, the reusable structures define these two objects:
erDiagram
"CustomTemperature(Enum)" {
val HOT "highly available"
val COLD "offline storage"
}
"CustomRating(Struct)" {
string customRatingFrom
long customRatingOf
}
If you do not need these kinds of structures in your model, you can leave these sections out entirely.
Define local variables for your types
To be capable of being referenced as types for attributes elsewhere in your model, the type names for these structures must be prefixed with the namespace. The toolkit provides a helper method for you to enforce this — getTypeName()
.
MyCustomModel.pkl | |
---|---|
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
- Define local variables (using the
local
keyword) for the name of each of your reusable structures. You can ensure they are properly namespaced by using thegetTypeName()
helper method the toolkit provides. - (Optional) You may define any number of lists of valid values that can be used to constrain values for some attribute elsewhere in your model within
customEnumTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
and a map ofvalidValues
for each enumeration. - The valid values should each be specified in the form
["Value"] { description = "" }
, where the value in square brackets is one acceptable value for this enumeration, and the description gives the meaning of that value. - (Optional) You may define any number of complex nested attribute structures that can be used as an attribute elsewhere in your model within
customStructTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
and a map ofattributes
for each struct. - Each attribute should take the form of
["name"] { }
and have at least adescription
andtype
. - The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute.
Define local variables for your types
To be capable of being referenced as types for attributes elsewhere in your model, the type names for these structures must be prefixed with the namespace. The toolkit provides a helper method for you to enforce this — getTypeName()
.
MyCustomModel.pkl | |
---|---|
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
|
- Define local variables (using the
local
keyword) for the name of each of your reusable structures. You can ensure they are properly namespaced by using thegetTypeName()
helper method the toolkit provides. - (Optional) You may define any number of lists of valid values that can be used to constrain values for some attribute elsewhere in your model within
customEnumTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
and a map ofvalidValues
for each enumeration. - The valid values should each be specified in the form
["Value"] { description = "" }
, where the value in square brackets is one acceptable value for this enumeration, and the description gives the meaning of that value. - (Optional) You may define any number of complex nested attribute structures that can be used as an attribute elsewhere in your model within
customStructTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
and a map ofattributes
for each struct. - Each attribute should take the form of
["name"] { }
and have at least adescription
andtype
. - The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute. - (Optional) Setting the
label
will control how the attribute is labelled in the user interface.
Define abstract supertype¶
(Optional) Use the supertypeDefinition
section to define reusable attributes for your model. This supertype itself would never be directly instantiated, but will define attributes that are common across all types that can be instantiated.
Running example (expand for details)
From the running example, the common metadata was not originally illustrated. If you noticed the same 3 attributes were defined again at each level, here this abstract type defines those attributes just once (to be inherited by all the other levels).
erDiagram
"Custom (Abstract)" {
string customSourceId
string customDatasetName
string customDatasetQualifiedName
}
Will inherit all attributes from its own supertype
Remember that this supertype will itself inherit all attributes from its supertype (by default, Catalog
). So things like name
, description
, qualifiedName
, createdBy
, updatedBy
, and so on do not need to be redefined here.
If you have no other attributes you need across the asset types in your model, you can leave this supertypeDefinition
out entirely and the toolkit will generate it for you.
MyCustomModel.pkl | |
---|---|
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
|
- You may define one (and only one)
supertypeDefinition
. - (Optional) Usually this should be the same as, or at least start with the type prefix (
namespace
). If unspecified, it will default to the string used fornamespace
. -
(Optional) You may specify an alternative supertype for your abstract type itself to extend. If unspecified, it will default to
Catalog
.Could vary depending on the supertype you want
The code shown here is for setting the supertype to
Catalog
. Because the toolkit would already default this value, it is necessary in this case to create an entirely new listing, to completely override the toolkit's defaults.If you want to instead use some other supertype entirely (like
BI
), you can simply use:37
supertypes { "BI" }
-
(Optional) You can define any number of attributes that should be inherited by all custom asset types in the model. Each attribute should take the form of
["name"] { }
and have at least adescription
andtype
. - The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute. - (Optional) You can also control how the attribute will be indexed. For example,
both
will create both an exact-match-usefulkeyword
index as well as a tokenized fuzzy-usefultext
index for that attribute. - (Optional) By default, string attributes will be indexed as a tokenized fuzzy-useful
text
index. If you want to force them to use an exact-match-usefulkeyword
index instead, you can set theindexAs
tokeyword
.
Why would I define these DatasetName
and DatasetQualifiedName
attributes as shared?
This is necessary to ensure these attributes exist on all asset types within this area, so that the hierarchy filters on the asset discovery UI find all children objects across all levels of the containment hierarchy.
This is a common pattern for new asset types that have a hierarchy of containment. You'll see the same pattern in our out-of-the-box SQL asset types, for example, which have databaseQualifiedName
, databaseName
, schemaQualifiedName
, schemaName
, tableQualifiedName
, tableName
, viewQualifiedName
, and viewName
all defined at the shared supertype level (SQL
).
MyCustomModel.pkl | |
---|---|
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
|
- You may define one (and only one)
supertypeDefinition
. - (Optional) Usually this should be the same as, or at least start with the type prefix (
namespace
). If unspecified, it will default to the string used fornamespace
. -
(Optional) You may specify an alternative supertype for your abstract type itself to extend. If unspecified, it will default to
Catalog
.Could vary depending on the supertype you want
The code shown here is for setting the supertype to
Catalog
. Because the toolkit would already default this value, it is necessary in this case to create an entirely new listing, to completely override the toolkit's defaults.If you want to instead use some other supertype entirely (like
BI
), you can simply use:40
supertypes { "BI" }
-
(Optional) You can define any number of attributes that should be inherited by all custom asset types in the model. Each attribute should take the form of
["name"] { }
and have at least adescription
andtype
. - The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute. - (Optional) Setting the
label
will control how the attribute is labelled in the user interface. - (Optional) You can also control how the attribute will be indexed. For example,
both
will create both an exact-match-usefulkeyword
index as well as a tokenized fuzzy-usefultext
index for that attribute. - (Optional) By default, string attributes will be indexed as a tokenized fuzzy-useful
text
index. If you want to force them to use an exact-match-usefulkeyword
index instead, you can set theindexAs
tokeyword
.
Why would I define these DatasetName
and DatasetQualifiedName
attributes as shared?
This is necessary to ensure these attributes exist on all asset types within this area, so that the hierarchy filters on the asset discovery UI find all children objects across all levels of the containment hierarchy.
This is a common pattern for new asset types that have a hierarchy of containment. You'll see the same pattern in our out-of-the-box SQL asset types, for example, which have databaseQualifiedName
, databaseName
, schemaQualifiedName
, schemaName
, tableQualifiedName
, tableName
, viewQualifiedName
, and viewName
all defined at the shared supertype level (SQL
).
Define instantiate-able types¶
Then, define the types in your custom model that you want to be able to instantiate.
Define local variables for your types and key attributes
To be capable of being referenced in relationships and in the generated UI code, both your type names and certain attributes must be prefixed with the namespace (or attribute prefix). The toolkit provides helper methods for you to get these — getTypeName()
and getAttributeName()
.
New types of assets¶
Describe the new types of assets you want to be able to create and manage (and their attributes) under the customAssetTypes
section.
Running example (expand for details)
From the running example, the new asset types define these three objects:
erDiagram
CustomDataset {
string customSourceId "from Custom"
string customDatasetName "from Custom"
string customDatasetQualifiedName "from Custom"
}
CustomTable {
string customSourceId "from Custom"
string customDatasetName "from Custom"
string customDatasetQualifiedName "from Custom"
struct[] customRatings
}
CustomField {
string customSourceId "from Custom"
string customDatasetName "from Custom"
string customDatasetQualifiedName "from Custom"
string tableName "from Column"
string tableQualifiedName "from Column"
enum customTemperature
}
"CustomTemperature(Enum)" |o--o{ CustomField : ""
"CustomRating(Struct)" |o--o{ CustomTable : ""
MyCustomModel.pkl | |
---|---|
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
- Define local variables (using the
local
keyword) for each of your types. You can ensure they are properly namespaced by using thegetTypeName()
helper method the toolkit provides. - You can define any number of custom types that can be instantiated within
customAssetTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
for that custom asset type.
MyCustomModel.pkl | |
---|---|
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
|
- Define local variables (using the
local
keyword) for each of your types. You can ensure they are properly namespaced by using thegetTypeName()
helper method the toolkit provides. - You can define any number of custom types that can be instantiated within
customAssetTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
for that custom asset type. - You can specify any attributes specific to this custom asset here. (Remember any common attributes will be inherited automatically from the supertype.)
- The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute. - The
type
can either be primitive or point to a complex definition likestruct
orenum
. -
When the
type
isstruct
, you must also provide the name of the struct instructName
.Use that local variable you created to define the struct!
-
(Optional) If you want to allow multiple instances of this attribute to be stored on each asset, set
multiValued
totrue
. - (Optional) You can specify any additional supertypes your custom type should have. The top-level supertype defined under
supertypeDefinition
(or generated from thenamespace
) will be set automatically, so you only need to include this if you want your new custom type to have multiple supertypes. - The
type
can beenum
to restrict its values to a set of predefined values. -
When the
type
isenum
, you must also provide the name of the enumeration that defines the valid values inenumName
.Use that local variable you created to define the enum!
MyCustomModel.pkl | |
---|---|
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 |
|
- Define local variables (using the
local
keyword) for each of your types. You can ensure they are properly namespaced by using thegetTypeName()
helper method the toolkit provides. - Define local variables (using the
local
keyword) for each attribute you will reference for the UI. You can ensure they are properly namespaced by using thegetAttributeName()
helper method the toolkit provides. - You may define further local variables to represent the various icons you want to use in the UI.
- Each icon must have a
name
that will be used to refer to it in the generated code. When you want to reuse an existing icon, this must match the name of the existing icon in the existing front-end code. - (Optional) Each icon can also have an alternate variation that is used when the icon is selected, which also must be named. (Again, if you want to reuse an existing icon, this must match the name of the existing icon in the existing front-end code.)
- Each icon must have an SVG file that provides the actual image for the icon. You'll need to copy this image file into the appropriate location later, but the filename must be accurate here.
- (Optional) When you want an alternate variation of the icon to use when the icon is selected, specify the SVG filename for that alternative image. (Again, you'll need to copy this image file into the appropriate location later, but the filename must be accurate here.)
- You can define any number of custom types that can be instantiated within
customAssetTypes
. - Use the local variables defined above in the form
[VariableName] { }
and provide at least adescription
for that custom asset type. - Setting the
label
will control how the type is labelled in the user interface. - Setting the
icon
will control the icon to display for this type in the user interface. These are themselves an object, which could either be defined inline here or (as in this example) as a separate local variable. -
For types that are contained within another type, specify the name of the de-normalized attribute that contains the
qualifiedName
of the parent asset. This will be used to efficiently render parent-child relationships in the UI.Use that local variable you created for the attribute!
-
You can specify any attributes specific to this custom asset here. (Remember any common attributes will be inherited automatically from the supertype.)
- The name of the attribute will automatically be prefixed with the attribute prefix (
attrPrefix
) for you, so you can focus on just a simple name for the attribute. - You can also set the
label
to control how each attribute is labelled in the user interface. - The
type
can either be primitive or point to a complex definition likestruct
orenum
. -
When the
type
isstruct
, you must also provide the name of the struct instructName
.Use that local variable you created to define the struct!
-
(Optional) If you want to allow multiple instances of this attribute to be stored on each asset, set
multiValued
totrue
. - (Optional) You can specify any additional supertypes your custom type should have. The top-level supertype defined under
supertypeDefinition
(or generated from thenamespace
) will be set automatically, so you only need to include this if you want your new custom type to have multiple supertypes. - The
type
can beenum
to restrict its values to a set of predefined values. -
When the
type
isenum
, you must also provide the name of the enumeration that defines the valid values inenumName
.Use that local variable you created to define the enum!
-
(Optional) You can also define UI aspects that should apply across all assets of these types under the
ui
section. - Provide the filename for an SVG image you want to use as the icon to visually present all of these assets. This could be a branded logo of the source system that these assets represent, for example.
-
(Optional) You can define the hierarchy of filters that users can apply on the asset discovery page.
- These are applicable in top-down order after a connection has been selected, and can include at most the top 2 levels of the asset containment hierarchy.
- Each entry should be keyed by the type name, and have as a value the name of the denormalized attribute that every asset must have populated to be contained within that level of the hierarchy.
-
Use those local variables again!
-
(Optional) You can define the "breadcrumb trail" that should be shown for assets to indicate their containment hierarchy.
- These define the breadcrumb in top-down order, and should generally not include more than 3 levels (or they will overrun the UI).
-
Each entry should be keyed by the type name, and have two values:
q
giving the name of the denormalized attribute that has the unique name of the asset this one is contained withina
giving the name of the denormalized attribute that has the simple name of the asset this one is contained within
New asset relationships¶
Describe the new relationships you want to be able to create and manage between assets under the customRelationshipTypes
section.
Running example (expand for details)
From the running example, the new relationships define these linkages between the assets:
erDiagram
CustomDataset ||--o{ CustomTable : "customTables / customDataset"
CustomTable ||--o{ CustomField : "customFields / customTable"
CustomField }o--o{ CustomField : "customToFields / customFromFields"
If you have no relationships you need between the new asset types in your model and any other asset types, you can leave this customRelationshipTypes section out entirely.
MyCustomModel.pkl | |
---|---|
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
|
-
You can also specify any number of custom relationships. These should be listed under
customRelationshipTypes
and each take the form of["name"] = new Relationship { }
. There are two kinds of Relationship that can be created:ContainmentRelationship
defines a parent-child (hierarchical) relationship. Each parent can have many children, but each child can refer to only a single parent.PeerToPeerRelationship
defines an association relationship, which is many-to-many by default.
Make the name as informational as you want
The name is not actually used other than to keep the relationships unique, so feel free to use as informational a name as you want.
-
This example starts by creating a new parent-child
ContainmentRelationship
between a dataset (the parent) and its tables (the children). - A
ContainmentRelationship
must have one and only oneparent
, which describes the parent end of the relationship. - Each end of the relationship must have a
type
, defining the asset type for that end of the relationship. - Each end of the relationship must also define an
attribute
, which is how this end of the relationship will be referred to by the other end of the relationship. The name of the attribute will automatically be prefixed with the attribute prefix (attrPrefix
) for you, so you can focus on just a simple name for the attribute. - Each end of the relationship must also provide a
description
for theattribute
. - A
ContainerRelationship
must also have one and only onechildren
definition, which describes the end of the relationship containing the children. - This example shows how you can define a new many-to-many relationship between assets.
- A
PeerToPeerRelationship
must have a listing of exactly twopeers
, each of which describes one end of the relationship. - Each end of the relationship must have a
type
, defining the asset type for that end of the relationship. - Each end of the relationship must also define an
attribute
, which is how this end of the relationship will be referred to by the other end of the relationship. The name of the attribute will automatically be prefixed with the attribute prefix (attrPrefix
) for you, so you can focus on just a simple name for the attribute. - Each end of the relationship must also provide a
description
for theattribute
.
MyCustomModel.pkl | |
---|---|
149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 |
|
-
You can also specify any number of custom relationships. These should be listed under
customRelationshipTypes
and each take the form of["name"] = new Relationship { }
. There are two kinds of Relationship that can be created:ContainmentRelationship
defines a parent-child (hierarchical) relationship. Each parent can have many children, but each child can refer to only a single parent.PeerToPeerRelationship
defines an association relationship, which is many-to-many by default.
Make the name as informational as you want
The name is not actually used other than to keep the relationships unique, so feel free to use as informational a name as you want.
-
This example starts by creating a new parent-child
ContainmentRelationship
between a dataset (the parent) and its tables (the children). - A
ContainmentRelationship
must have one and only oneparent
, which describes the parent end of the relationship. - Each end of the relationship must have a
type
, defining the asset type for that end of the relationship. - Each end of the relationship must also define an
attribute
, which is how this end of the relationship will be referred to by the other end of the relationship. The name of the attribute will automatically be prefixed with the attribute prefix (attrPrefix
) for you, so you can focus on just a simple name for the attribute. - Each end of the relationship must also provide a
description
for theattribute
. - A
ContainerRelationship
must also have one and only onechildren
definition, which describes the end of the relationship containing the children. - This example shows how you can define a new many-to-many relationship between assets.
- A
PeerToPeerRelationship
must have a listing of exactly twopeers
, each of which describes one end of the relationship. - Each end of the relationship must have a
type
, defining the asset type for that end of the relationship. - Each end of the relationship must also define an
attribute
, which is how this end of the relationship will be referred to by the other end of the relationship. The name of the attribute will automatically be prefixed with the attribute prefix (attrPrefix
) for you, so you can focus on just a simple name for the attribute. - Each end of the relationship must also provide a
description
for theattribute
.
Advanced attribute options¶
There are further advanced options you can use when defining each attribute. These will all be set to sensible defaults based on the options outlined in the example above, but you can also directly set or override them if needed.
Property | Usage | Default |
---|---|---|
defaultValue |
Default value for the attribute. | |
isDefaultValueNull |
Indicates whether the attribute has a default value of being empty (true) or not (false). | |
isOptional |
Indicates whether the attribute is mandatory (false) or optional (true). | true |
valuesMinCount |
Minimum number of values the attribute should have. | |
valuesMaxCount |
Maximum number of values the attribute should have. | |
isUnique |
Whether the attribute is unique (true) or not (false). | false |
isIndexable |
Whether the attribute is indexed in-memory via Cassandra (true) or not (false). | false |
includeInNotification |
Whether the attribute should generate a notification when its value changes (true) or not (false). | true |
skipScrubbing |
TBC | true |
searchWeight |
TBC | |
indexAs |
What kind of index(es) to create in Elastic for this attribute: keyword , text , both . |
default |