Uploaded image for project: 'mod-search'
  1. mod-search
  2. MSEARCH-349

Change Instance ids stream API to use jobs

    XMLWordPrintable

Details

    • Task
    • Status: Closed (View Workflow)
    • P3
    • Resolution: Done
    • None
    • None
    • Spitfire Sprint 145
    • 2
    • Spitfire
    • Nolana R3 2022

    Description

       Requirements/Scope:

      1. Extend mod-search.yaml with new API /instance/ids/jobs
      2. Add unit/integration tests

      Acceptance criteria:

      • New API for /instance/ids/jobs is exist
      • Unit/Integration tests created

      Proposed solution:
      A possible solution for streaming large volumes of IDs. To begin with, it is worth dividing streaming into two operations. Stream ID from the ES to temporary storage, and the next step is to stream unique IDs to the end-user. As temporary storage, it is most convenient to use the Postgres database.
      At request, it is possible to create a temporary table in which to add the IDs from the ES. To optimize export, name the table with the CQL query hash and reuse it. A periodic job can delete result tables after it is no longer needed.

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                Shans Shans Kaluhin
                OleksiiKuzminov Oleksii Kuzminov
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases