Uploaded image for project: 'data-import-processing-core'
  1. data-import-processing-core
  2. MODDICORE-200

Overlaying with single record import creates duplicate control fields on Juniper bugfest - KIWI BF

    XMLWordPrintable

    Details

    • Template:
    • Story Points:
      3
    • Development Team:
      Folijet Support
    • Release:
      R3 2021 Bug Fix
    • Hot Fix Approved by Cap Planning?:
      Yes
    • Hot Fix Approval Comments:
      Approved by CPT on Slack release_bug_triage channel on 22 Oct 2021 as Juniper HF3; changed to Kiwi BF 26 Oct 2021. See additional info in comments below.

      Description

      Overview: MARC Bib control fields (005, 006, 007, 008) are being duplicated when the SRS MARC Bibs record is overlaid/updated via Inventory Single Record Import or via Data Import. quickMARC is not affected

      NOTE: In UIDATIMP-1043, we have restricted the Ind 1, Ind 2, and Subfield so that they cannot be assigned to a MARC LDR, 001-009 control fields. See that Jira for additional details.

      Current Workaround: do not include the 005, 006, 007, 008 control fields as protected fields in Data Import settings

      I think the confusion and complexity is because:

      1. MARC control fields (Leader, 001-009) do not have indicators or subfields, but variable fields do
      2. Some MARC control fields are system controlled or calculated
      3. Some MARC control fields are non-repeatable, but some are repeatable
      4. If a field is repeatable, and is protected, then any unique versions of that field in the incoming record should be added, but any duplicate versions of that field should not be added
      5. These protections only apply when a MARC Update action is included in the Job profile (currently)
      6. If a MARC update happens implicitly due to an Instance Update action, then field protections do not apply, and all fields in the SRS MARC are updated (except special handling for 001 and 999 ff)
      7. We need to resolve this well enough to release Kiwi without duplicate, repeatable control fields (006s and 007s) being created. More complete fixing will be specified, analyzed carefully by SMEs, QA, and Devs, and then implemented in Lotus

      Steps to Reproduce:

      1. Log into Juniper bugfest (abreaux/admin)
      2. Go to Inventory and search for HRID ak00005006874
      3. Display the instance detail screen and select Actions/View Source
      4. Note how many 005, 006, 007, and 008 fields there are in the MARC record
      5. Then close View source and select Actions/Overlay
      6. In the Single record import pop-up, select OCLC WorldCat and enter 36865429 as the identifier
      7. Once the record has been updated, select Actions/View source
      8. Check to see if there are multiple 005s, 006s, 007s, 008s

      NOTE

      • The same bug is seen when an existing Instance is updated with a regular Data Import profile (e.g. job profile 001-001 marc match smoke test
      • This only happens in Juniper Bugfest (with the Hotfix 3 changes), not in any Juniper environment with HF 1 and 2, and not in Kiwi

      Expected Results:
      The existing record is replaced with the OCLC record

      Actual Results:
      005, 006, 007, and 008 from the previous record are retained resulting in double fields

      NOTE: This was caused by a field protection setting of Field = *, Ind 1/2 = *, Subfield = 5, Data = *
      For the control fields:

      • LDR, 001, 003, 005, 008 are non-repeatable
      • 006, 007 are repeatable
      • What happens during updates depends on which fields are protected, whether that field is repeatable or not, and for repeatable fields, whether the incoming data is the same as the existing data or not

      Example 1

      • Field protection setting = Field = *, Ind 1/2 = *, Subfield = 5, Data = *
      • Existing record has following control fields
        • 001: in001
        • 008: 123
        • 006: abc
        • 006: def
      • Overlay is performed by a record with these fields
        • 001: ybp74
        • 008: 456
        • 006: xyz
      • Because 001 and 008 are non-repeatable fields (or 003 or 005 or Leader), and that field is protected in the field protection settings, then the 001 and 008 in the incoming record should both be discarded
      • If it's a repeatable field (006, 007), the incoming field should be added as an additional one, without deleting any existing ones, except the added fields should not be a duplicate to an existing field
      • So the updated record would have
        • 001: in001
        • 008: 123
        • 006: abc
        • 006: def
        • 006: xyz

      Example 2

      • Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
      • Existing record has following control fields
        • 001: in001
        • 008: 123
        • 006: abc
        • 006: def
      • Overlay is performed by a record with these fields
        • 001: ybp74
        • 008: 456
        • 006: xyz
      • Because 008 is non-repeatable field (or 003 or 005 or Leader), and that field is NOT protected in the field protection settings, then the 008 in the incoming record should replace the existing 008
      • If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should be added, but only if the data in it is different from the data in that field in the existing record. Since the incoming 006 is different, it is added, and the updated record ends up with three 006 fields.
      • So the updated record would have
        • 001: in001
        • 008: 456
        • 006: abc
        • 006: def
        • 006: xyz

      Example 3

      • Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
      • Existing record has following control fields
        • 001: in001
        • 006: abc
        • 006: def
      • Overlay is performed by a record with these fields
        • 001: ybp74
        • 006: abc
        • 006: xyz
      • If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should only be added if the data in it is different from the data in that field in the existing record.
      • So the updated record would have the two 006s from the existing record, plus the unique 006 from the incoming record, but would not have the duplicate 006 from the incoming record (abc)
        • 001: in001
        • 006: abc
        • 006: def
        • 006: xyz

      Example 4

      • Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
      • Existing record has following control fields
        • 001: in001
        • 006: abc
        • 006: def
      • Overlay is performed by a record with these fields
        • 001: ybp74
        • 006: abc
      • If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should only be added if the data in it is different from the data in that field in the existing record. In this case, the incoming 006 is the same as an existing 006
      • So the updated record would have
        • 001: in001
        • 006: abc
        • 006: def

      Example 5

      • No field protection settings, or no Update MARC Bib action in Job Profile
      • Existing record has following control fields
        • 001: in001
        • 006: abc
        • 006: def
      • Overlay is performed by a record with these fields
        • 001: ybp74
        • 006: abc
        • 006: xyz
      • Because there are no field protections, all fields in the SRS MARC record (except 001 and 999 ff) are replaced with the incoming MARC data
      • So the updated record would have the 001 from the existing record, and the 006s from the incoming record
        • 001: in001
        • 006: abc
        • 006: xyz

      Additional Juniper records that can be used for reproducing and testing:
      (HRID is first, then the OCLC number)
      ak00005007612 42980246
      in7023981 42980246
      ins00007267522 42980246
      ak00005007664 43311780
      ins00007267543 43311780
      in7022003 43311780
      in8779566 43311780
      ak00005507053 36485985
      ak00005003633 36485985
      ak00005275045 36485985
      ak00005007101 42668854
      in7022076 42668854
      ins00007266968 42668854

      Interested parties:

      Charlotte Whitt Ann-Marie Breaux Khalilah Gambrell

        TestRail: Results

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                ruslan_lavrov Ruslan Lavrov
                Reporter:
                jenncolt Jenn Colt
                Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases