Paging search results¶

Automatically (via SDK)¶

Our SDKs are designed to simplify paging, so you do not need to worry about the underlying details. You can simply iterate through a search response and the SDK will automatically fetch the next page(s) when it needs to (lazily).

The SDKs will even add a default sort by GUID to ensure stable results across pages, even when you do not provide any sorting criteria yourself.

Java Python Kotlin Raw REST API

Automatic paging

client.assets.select() // You can start building a query across all assets using the select() method on the assets member of any client. You can chain as many mandatory (where()) conditions, mandatory exclusion (whereNot()) conditions, and set of conditions some of which must match (whereSome()) as you want.

    .pageSize(50) // The number of results to include (per page).

    .stream() // 
You can stream the results direct from the response. This will also lazily load and loop through each page of results.

Can be chained without creating a request in-between
You can actually chain the stream() method directly onto the end of your query and request construction, without creating a request or response object in-between.


    .limit(100) // 
With streaming, you can apply your own limits to the maximum number of results you want to process.

Independent of page size
Note that this is independent of page size. You could page through results 50 at a time, but only process a maximum of 100 total results this way. Since the results are lazily-loaded when streaming, only the first two pages of results would be retrieved in such a scenario.


    .filter(a -> !(a instanceof ILineageProcess)) // 
You can also apply your own logical filters to the results.

Push-down as much as you can to the query
You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that cannot be encoded in the query this can be a useful secondary filter over the results.


    .forEach(a -> log.info("Do something with each result: {}", a)); // 
The forEach() on the resulting stream will then apply whatever actions you want with the results that come through.

Build the query

from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import Asset, Process
from pyatlan.model.fluent_search import CompoundQuery, FluentSearch

builder = (
    FluentSearch()  # You can start building a query using a FluentSearch object. You can have as many mandatory (where()) conditions, mandatory exclusion (where_not()) conditions, and set of conditions some of which must match (where_some()) as you want.

    .where(CompoundQuery.active_assets())  # This helper provides a query that ensures results are active (not archived) assets.

).to_request()  # You can now build all of this search configuration into a request.

results = client.asset.search(index)  # You can then run the search against this request.

for asset in results: # 
This will iterate through all the results without the need to be concerned with pages.

Iterating over results produces a Generator
This means that results are retrieved from the backend a page at time. This also means that you can only iterate over the results once.


    if not isinstance(asset, Process): # 
Remember that each result is a generic Asset. You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that cannot be encoded in the query this can be a useful secondary filter over the results.

        non_process = asset

Automatic paging

client.assets.select() // You can start building a query across all assets using the select() method on the assets member of any client. You can chain as many mandatory (where()) conditions, mandatory exclusion (whereNot()) conditions, and set of conditions some of which must match (whereSome()) as you want.

    .pageSize(50) // The number of results to include (per page).

    .stream() // 
You can stream the results direct from the response. This will also lazily load and loop through each page of results.

Can be chained without creating a request in-between
You can actually chain the stream() method directly onto the end of your query and request construction, without creating a request or response object in-between.


    .limit(100) // 
With streaming, you can apply your own limits to the maximum number of results you want to process.

Independent of page size
Note that this is independent of page size. You could page through results 50 at a time, but only process a maximum of 100 total results this way. Since the results are lazily-loaded when streaming, only the first two pages of results would be retrieved in such a scenario.


    .filter { it !is ILineageProcess } // 
You can also apply your own logical filters to the results.

Push-down as much as you can to the query
You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that cannot be encoded in the query this can be a useful secondary filter over the results.


    .forEach { log.info { "Do something with each result: $it" } } // 
The forEach() on the resulting stream will then apply whatever actions you want with the results that come through.

Use an SDK

The SDKs manage making multiple requests and parsing results to make subsequent requests in the most efficient way possible. You will need to make many different API requests if you want to do the same directly via the raw REST APIs.

Manually (via Elastic)¶

For curious minds, though, you can page through search results using a combination of the following properties¹:

Property	Description	Example
`from`	Indicates the starting point for the results.	`0`
`size`	Indicates how many results to include per response (page). As a general rule of thumb we would recommend a size from `20`-`100`, making `50` a common starting point.	`50`
`track_total_hits`	Includes an accurate number of total results, if set to `true`. With its default value on the raw REST APIs (`false`) the maximum number of results you will see in the `approximateCount` field in the response is 10000. (Again, the SDKs set this to `true` by default to avoid this confusion.)	`true`

Constraints with this approach

To have the most consistent results you can when paging, you must always use some sorting criteria and include at least one sorting criteria as a tie-breaker. (You must also keep that criteria the same for every page.)

Furthermore, as you get to larger from sizes (more than ~10,000) Elastic will begin to use significantly more resources to process your paging. To reduce this impact, if you need to page through many results you should implement your own timestamp-based offset mechanism so that the from size is kept consistently low.

(Again, the SDKs do both of these for you automatically.)

1.4.0 1.1.0

For example:

Java Python Kotlin Raw REST API

Annotated sort options, as you would define them in the Java SDK
SortOptions byUpdate = Asset.UPDATE_TIME.order(SortOrder.Desc); // Include any of your own sorting, like this example putting the most recently-updated assets first in the results.

SortOptions byGuid = Asset.GUID.order(SortOrder.Asc); // Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

Build the request
IndexSearchRequest index = IndexSearchRequest.builder(
  IndexSearchDSL.builder(someQuery) // You still need a query, to get some results .

      .from(100) // Starting point for the page of results being requested. In this example, you would be asking for the third page. (0 would be from 0-50 for the first page, 50 would be from 50-100 for the second page, and this gives us 100-150 for the third page.)

      .size(50) // The number of results per page (in this example, 50 results per page).

      .trackTotalHits(true) // Enable trackTotalHits so that your response includes an accurate total number of results. (Actually the Java SDK enables this by default, so this step is redundant unless you want to turn it off.)

      .sortOption(byUpdate) // And we need to include the sorting criteria we defined just above.

      .sortOption(byGuid)
      .build())
    .build();

Iterate through multiple pages of results
IndexSearchResponse response = index.search(client); // Keep the response object from the initial search, as it has a helper method for paging.

long totalResults = response.getApproximateCount(); // Since we set trackTotalHits to true (the default for the Java SDK even if we do not set it), the .getApproximateCount() will give us the total number of results. This can be over 10,000.

for (Asset result : response) { // Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results).

    // Do something with each result of the search...
}
response.forEach(a -> log.info("Found asset: {}", a.getGuid())); // Alternatively, you can iterate through all the results using forEach() on the response. (This uses the same underlying iterable-based implementation.)

response.stream() // Alternatively, you can stream the results. Streaming will also lazily-load only the pages of results necessary to meet the chained criteria for processing the stream.

    .filter(a -> !(a instanceof ILineageProcess)) // When streaming, you can further filter the results to apply any complex filtering logic you could not push-down as part of the query itself.

    .limit(100) // When streaming, you can also limit the total number of results you want to process — independently of the page size.

    .forEach(a -> log.info("Found asset: {}", a.getGuid())) // Don't forget to actually do something with the results in the stream 

Annotated sort options, as you would define them in the Python SDK
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.enums import SortOrder
from pyatlan.model.assets import Referenceable
from pyatlan.model.search import IndexSearchRequest, DSL

by_update = Referenceable.UPDATE_TIME.order(SortOrder.DESCENDING)  # Include any of your own sorting, like this example putting the most recently-updated assets first in the results.

by_guid = Referenceable.GUID.order(SortOrder.ASCENDING)  # Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

Build the request
index = IndexSearchRequest(
    dsl=DSL(
        query=someQuery,  # You still need a query, to get some results .

        from_=100,  # Starting point for the page of results being requested. In this example, you would be asking for the third page. (0 would be from 0-50 for the first page, 50 would be from 50-100 for the second page, and this gives us 100-150 for the third page.)

        size=50,  # The number of results per page (in this example, 50 results per page).

        track_total_hits=True,  # Enable track_total_hits so that your response includes an accurate total number of results. (Actually the Python SDK enables this by default, so this step is redundant unless you want to turn it off.)

        sort=[  # And we need to include the sorting criteria we defined just above.

            by_update,
            by_guid
        ],
    )
)

Iterate through multiple pages of results
client = AtlanClient()
response = client.asset.search(index)  # Keep the response object from the initial search, as it has a helper method for paging.

total_results = response.count  # Since we set track_total_hits to True (the default for the Python SDK even if we do not set it), the .count property will give us the total number of results. This can be over 10,000.

for result in response:  # Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results). Don't forget to actually do something with the results in the stream 

    # Do something with each result of the search...

Annotated sort options, as you would define them in the Java SDK
val byUpdate = Asset.UPDATE_TIME.order(SortOrder.Desc) // Include any of your own sorting, like this example putting the most recently-updated assets first in the results.

val byGuid = Asset.GUID.order(SortOrder.Asc) // Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

Build the request
val index = IndexSearchRequest.builder(
  IndexSearchDSL.builder(someQuery) // You still need a query, to get some results .

      .from(100) // Starting point for the page of results being requested. In this example, you would be asking for the third page. (0 would be from 0-50 for the first page, 50 would be from 50-100 for the second page, and this gives us 100-150 for the third page.)

      .size(50) // The number of results per page (in this example, 50 results per page).

      .trackTotalHits(true) // Enable trackTotalHits so that your response includes an accurate total number of results. (Actually the Java SDK enables this by default, so this step is redundant unless you want to turn it off.)

      .sortOption(byUpdate) // And we need to include the sorting criteria we defined just above.

      .sortOption(byGuid)
      .build())
    .build()

Iterate through multiple pages of results
val response = index.search(client) // Keep the response object from the initial search, as it has a helper method for paging.

val totalResults = response.approximateCount // Since we set trackTotalHits to true (the default for the Java SDK even if we do not set it), the .getApproximateCount() will give us the total number of results. This can be over 10,000.

for (result in response) { // Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results).

    // Do something with each result of the search...
}
response.forEach { log.info { "Found asset: ${it.guid}" } } // Alternatively, you can iterate through all the results using forEach() on the response. (This uses the same underlying iterable-based implementation.)

response.stream() // Alternatively, you can stream the results. Streaming will also lazily-load only the pages of results necessary to meet the chained criteria for processing the stream.

    .filter { it !is ILineageProcess } // When streaming, you can further filter the results to apply any complex filtering logic you could not push-down as part of the query itself.

    .limit(100) // When streaming, you can also limit the total number of results you want to process — independently of the page size.

    .forEach { log.info { "Found asset: ${it.guid}" } } // Don't forget to actually do something with the results in the stream 

POST /api/meta/search/indexsearch
{
  "dsl": {
    "from": 100, // Starting point for the page of results being requested. In this example, you would be asking for the third page. (0 would be from 0-50 for the first page, 50 would be from 50-100 for the second page, and this gives us 100-150 for the third page.)

    "size": 50, // The number of results per page (in this example, 50 results per page).

    "track_total_hits": true, // Enable track_total_hits so that your response includes an accurate total number of results.

    "query": {...}, // You still need a query, to get some results .

    "sort": [ // When paging, we should always sort the results (for consistency across the pages).

      { "__modificationTimestamp": { "order": "desc" }}, // Include any of your own sorting, like this example putting the most recently-updated assets first in the results.

      { "__guid": { "order": "asc" }} // Also consider a tie-breaker sorting mechanism. In this case, we use the GUID of an asset to further sort any results that have the same last modified timestamp.

    ]
  }
}

Annotated response, in plain JSON
{
  "queryType": "INDEX",
  "searchParameters": {
      "showSearchScore": false,
      "suppressLogs": false,
      "allowDeletedRelations": false,
      "query": "{\"from\":100,\"size\":50,\"track_total_hits\":true,\"query\":{...},\"sort\":[{\"__modificationTimestamp\":\"asc\"},{\"__guid\":\"asc\"}]}" // Note that every response to a search includes the query that was run. You can deconstruct this programmatically to always determine the from you will need for the next page of results. (Basically: from = from + size.) And in fact, since you can programmatically extract both the query and sorting criteria from this you have everything you need to get the next page — the query, the from, the size and the sort.

  },
  "entities": [ // The results themselves are the objects in the entities array. The size of this array will be at most size elements. Of course, your final page of results may not have a complete page of results, so it is possible that this array will be less than size (in particular, when you are on the final page).

    {...},
    {...},
    ...
  ],
  "approximateCount": 24631 // Since the request sets track_total_hits to true, the approximateCount in the response will have an accurate number of total results. Note that this can go beyond 10,000.

}

If you're familiar with Elasticsearch there are an alternative paging options using search_after and point-in-time (PIT) state preservation. (There also used to be scrolling, but this is no longer recommended by Elasticsearch.) We do not currently expose the search_after or PIT approaches through Atlan's search. However, you should still be able to page beyond the first 10,000 results using the approach outlined above. ↩