Athena assets package

The Athena assets package crawls Athena assets and publishes them to Atlan for discovery.

Direct extraction

Will create a new connection

This should only be used to create the workflow the first time. Each time you run this method it will create a new connection and new assets within that connection — which could lead to duplicate assets if you run the workflow this way multiple times with the same settings.

Instead, when you want to re-crawl assets, re-run the existing workflow (see Re-run existing workflow below).

1.0.0

To crawl Athena assets directly from Athena:

Java Python Raw REST API

Workflow athena = AthenaCrawler.iamUserAuth( // (1)
                "production", // (2)
                "athena.ap-south-1.amazonaws.com", // (3)
                443, // (4)
                "pEBDcOauyVfCtQacdvJUL", // (5)
                "eKnUIFK2JvmVO5mruqbxjnzXf", // (6)
                "s3://bucket/location", // (7)
                "workgroup", // (8)
                List.of(RoleCache.getIdForName("$admin")), // (8)
                null,
                null,
                true, // (9)
                true, // (10)
                10000L, // (11)
                Map.of("AwsDataCatalog", List.of("schema_one", "schema_two")), // (12)
                null); // (13)
WorkflowResponse response = athena.run(); // (14)

1. The AthenaCrawler package will create a workflow to crawl assets from Athena. The iamUserAuth() method creates a workflow for crawling assets directly from Athena using IAM-based authentication.
2. You must provide a name for the connection that the Athena assets will exist within.
3. You must provide the hostname of your Athena instance.
4. You must specify the port number of the Athena instance (use 443 for the default).
5. You must provide your IAM access key for accessing Athena.
6. You must provide your IAM secret key for accessing Athena.
7. You must provide an S3 location these IAM credentials can access for storing Athena query results.

8. You must specify at least one connection admin, either:

   • everyone in a role (in this example, all $admin users)
   • a list of groups (names) that will be connection admins
   • a list of users (names) that will be connection admins

9. You can also specify whether to allow queries against these Athena assets (true) or disable all querying to these assets (false).

10. You can also specify whether to allow data in these Athena assets to be previewed (true) or not (false).
11. You can also specify the maximum number of rows you'll allow to be returned from a query against these Athena assets.
12. You can also optionally specify the list of schemas to include in crawling. For Athena assets, this should be specified as a list of schema names provided as the value in a map with the key AwsDataCatalog. (If set to null, all schemas will be crawled.)
13. You can also optionally specify the list of schemas to exclude from crawling. For Athena assets, this should be specified as a list of schema names provided as the value in a map with the key AwsDataCatalog. (If set to null, no schemas will be excluded.)

14. You can then run the workflow using the run() method on the object you've created.

    Workflows run asynchronously

    Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed.

Coming soon

Create the workflow via UI only

We recommend creating the workflow only via the UI. To rerun an existing workflow, see the steps below.

Re-run existing workflow

1.10.6

To re-run an existing workflow for Athena assets:

Java Python Raw REST API

List<WorkflowSearchResult> existing = WorkflowSearchRequest // (1)
            .findByType(AthenaCrawler.PREFIX, 5); // (2)
// Determine which of the results is the Athena workflow you want to re-run...
WorkflowRunResponse response = existing.get(n).rerun(); // (3)

1. You can search for existing workflows through the WorkflowSearchRequest class.
2. You can find workflows by their type using the findByType() helper method and providing the prefix for one of the packages. In this example, we do so for the AthenaCrawler. (You can also specify the maximum number of resulting workflows you want to retrieve as results.)

3. Once you've found the workflow you want to re-run, you can simply call the rerun() helper method on the workflow search result. The WorkflowRunResponse is just a subtype of WorkflowResponse so has the same helper method to monitor progress of the workflow run.

   • Optionally, you can use the rerun(true) method with idempotency to avoid re-running a workflow that is already in running or in a pending state. This will return details of the already running workflow if found, and by default, it is set to false

   Workflows run asynchronously

   Remember that workflows run asynchronously. See the packages and workflows introduction for details on how you can check the status and wait until the workflow has been completed.

Coming soon

Requires multiple steps through the raw REST API

1. Find the existing workflow.
2. Send through the resulting re-run request.

{
  "from": 0,
  "size": 5,
  "query": {
    "bool": {
      "filter": [
        {
          "nested": {
            "path": "metadata",
            "query": {
              "prefix": {
                "metadata.name.keyword": {
                  "value": "atlan-athena" // (1)
                }
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "metadata.creationTimestamp": {
        "nested": {
          "path": "metadata"
        },
        "order": "desc"
      }
    }
  ],
  "track_total_hits": true
}

1. Searching by the atlan-athena prefix will ensure you only find existing Athena assets workflows.

   Name of the workflow

   The name of the workflow will be nested within the _source.metadata.name property of the response object. (Remember since this is a search, there could be multiple results, so you may want to use the other details in each result to determine which workflow you really want.)

{
  "namespace": "default",
  "resourceKind": "WorkflowTemplate",
  "resourceName": "atlan-athena-1684500411" // (1)
}

1. Send the name of the workflow as the resourceName to rerun it.

---