Spark¶

These are the model elements in Atlan related to Spark.

Reference documentation

This is reference documentation covering the entire Atlan model. It is not the best place to start when trying to understand how to manage Spark assets in Atlan. For that, we would suggest starting with either:

Snippets — small, atomic examples of single-step use cases.
Patterns — walkthroughs of common multi-step implementation patterns.

classDiagram
    direction RL
    class Spark {
        <<abstract>>
    }
    link Spark "../spark"
    class Catalog {
        <<abstract>>
    }
    link Catalog "../catalog"
    Catalog <|-- Spark : extends
    class Asset {
        <<abstract>>
    }
    link Asset "../asset"
    Asset <|-- Catalog : extends
    class Referenceable {
        <<abstract>>
    }
    link Referenceable "../referenceable"
    Referenceable <|-- Asset : extends
    class SparkJob
    link SparkJob "../entities/sparkjob"
    Spark <|-- SparkJob : extends

Properties¶

These attributes are specific to instances of Spark (and all of its subtypes).

sparkRunEndTime ¶

End time of the Spark Job eg. 1695673598218

sparkRunOpenLineageState ¶

OpenLineage state of the Spark Job run eg. COMPLETE

sparkRunOpenLineageVersion ¶

OpenLineage Version of the Spark Job run eg. 1.1.0

sparkRunStartTime ¶

Start time of the Spark Job eg. 1695673598218

sparkRunVersion ¶

Spark Version for the Spark Job run eg. 3.4.1

Relationships¶

The following illustrates how the various pieces of Azure Service Bus inter-relate with each other:

erDiagram
    Connection ||..o{ Process : ""
    Process ||--o{ SparkJob : sparkJobs
    Asset }o--o{ SparkJob : inputs
    Asset }o--o{ SparkJob : outputs