Uploaded image for project: 'mod-kb-ebsco-java'
  1. mod-kb-ebsco-java
  2. MODKBEKBJ-255

SPIKE - Harden process to load holdings

    XMLWordPrintable

    Details

    • Template:
    • Sprint:
      eHoldings Sprint 66
    • Story Points:
      5
    • Development Team:
      Spitfire

      Description

      We have a process in place to load holdings into our DB table - make requests to RM API holdings endpoints and populate holdings table in our DB. This process uses the https://github.com/folio-org/okapi/blob/master/doc/guide.md#timer interface and populates data every 2 days. This process is fragile in the sense that anything could go wrong in hitting the RM API endpoints or populating our database and we need to figure out how we can harden the process.

      Few approaches that we discussed:
      1. Create a separate endpoint that gives us the status of the progress of the process.
      2. Add retry mechanism in case something fails - at least 3 retries
      3. What happens if someone tries to filter resources by tags while the table is being populated/has no entries - we end up making multiple requests to RM API - which can be avoided if we add an updated_at column to our holdings table. Based on when the entry was last updated, we decide whether we want to insert/update the entry or leave it as is - this eliminates the need to truncate the entire holdings table and re-populate increasing performance.
      4. What happens if RM API is down? Approach outlined in 3. helps us still retain holdings data albeit stale.

        TestRail: Results

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                Natalia Zaitseva Natalia Zaitseva
                Reporter:
                sduvvuri Sobha Duvvuri
                Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases