Uploaded image for project: 'mod-user-import'
  1. mod-user-import
  2. MODUIMP-4

Bulk import of users needs performance improvements

    XMLWordPrintable

    Details

    • Template:

      Description

      We need performant bulk import of users for both migration and ongoing operations.

      The current user import module submits users one-by-one, which is rather slow at a scale of, say, 90K users, especially when trying to test data loading at scale. SysOps foresees similar issues with loading data into other modules.

      There are two general scenarios:

      1. At migration time, sites will need to do an initial bulk import from their existing systems, and new UUIDs will somehow need to be minted as part of the migration process.
      2. In ongoing operations, sites will do regular bulk updates from their campus identity management (IdM) infrastructure to refresh their campus users. In this case, existing UUIDs and some internal FOLIO fields will need to be preserved.

      The migration scenario here seems like a special case of the bulk import to support operations.

      There will be a need for bulk imports across FOLIO modules for both migration and operations. Users seems like a good starting point to consider some common strategies or techniques for bulk import that could inform best practices for project as a whole.

      This is related to MODUSERS-3.

        TestRail: Results

          Attachments

          1. users.zip
            4.42 MB
          2. vagrant_top.png
            vagrant_top.png
            95 kB

            Issue Links

              Activity

                People

                Assignee:
                Unassigned Unassigned
                Reporter:
                tod Tod Olson
                Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                  Dates

                  Created:
                  Updated:

                    TestRail: Runs

                      TestRail: Cases