Overlaying with single record import creates duplicate control fields on Juniper bugfest - LOTUS

Overview: MARC Bib control fields (005, 006, 007, 008) are being duplicated when the SRS MARC Bibs record is overlaid/updated via Inventory Single Record Import or via Data Import. quickMARC is not affected

NOTE: In UIDATIMP-1043, we have restricted the Ind 1, Ind 2, and Subfield so that they cannot be assigned to a MARC LDR, 001-009 control fields. See that Jira for additional details.

Current Workaround: do not include the 005, 006, 007, 008 control fields as protected fields in Data Import settings

I think the confusion and complexity is because:

1. MARC control fields (Leader, 001-009) do not have indicators or subfields, but variable fields do
2. Some MARC control fields are system controlled or calculated
3. Some MARC control fields are non-repeatable, but some are repeatable
4. If a field is repeatable, and is protected, then any unique versions of that field in the incoming record should be added, but any duplicate versions of that field should not be added
5. These protections only apply when a MARC Update action is included in the Job profile (currently)
6. If a MARC update happens implicitly due to an Instance Update action, then field protections do not apply, and all fields in the SRS MARC are updated (except special handling for 001 and 999 ff)
7. We need to resolve this well enough to release Kiwi without duplicate, repeatable control fields (006s and 007s) being created. More complete fixing will be specified, analyzed carefully by SMEs, QA, and Devs, and then implemented in Lotus

Steps to Reproduce:

1. Log into Juniper bugfest (abreaux/admin)
2. Go to Inventory and search for HRID ak00005006874
3. Display the instance detail screen and select Actions/View Source
4. Note how many 005, 006, 007, and 008 fields there are in the MARC record
5. Then close View source and select Actions/Overlay
6. In the Single record import pop-up, select OCLC WorldCat and enter 36865429 as the identifier
7. Once the record has been updated, select Actions/View source
8. Check to see if there are multiple 005s, 006s, 007s, 008s

NOTE

• The same bug is seen when an existing Instance is updated with a regular Data Import profile (e.g. job profile 001-001 marc match smoke test
• This only happens in Juniper Bugfest (with the Hotfix 3 changes), not in any Juniper environment with HF 1 and 2, and not in Kiwi

Expected Results:
The existing record is replaced with the OCLC record

Actual Results:
005, 006, 007, and 008 from the previous record are retained resulting in double fields

NOTE: This was caused by a field protection setting of Field = *, Ind 1/2 = *, Subfield = 5, Data = *
For the control fields:

• LDR, 001, 003, 005, 008 are non-repeatable
• 006, 007 are repeatable
• What happens during updates depends on which fields are protected, whether that field is repeatable or not, and for repeatable fields, whether the incoming data is the same as the existing data or not

Example 1

• Field protection setting = Field = *, Ind 1/2 = *, Subfield = 5, Data = *
• Existing record has following control fields
  • 001: in001
  • 008: 123
  • 006: abc
  • 006: def
• Overlay is performed by a record with these fields
  • 001: ybp74
  • 008: 456
  • 006: xyz
• Because 001 and 008 are non-repeatable fields (or 003 or 005 or Leader), and that field is protected in the field protection settings, then the 001 and 008 in the incoming record should both be discarded
• If it's a repeatable field (006, 007), the incoming field should be added as an additional one, without deleting any existing ones, except the added fields should not be a duplicate to an existing field
• So the updated record would have
  • 001: in001
  • 008: 123
  • 006: abc
  • 006: def
  • 006: xyz

Example 2

• Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
• Existing record has following control fields
  • 001: in001
  • 008: 123
  • 006: abc
  • 006: def
• Overlay is performed by a record with these fields
  • 001: ybp74
  • 008: 456
  • 006: xyz
• Because 008 is non-repeatable field (or 003 or 005 or Leader), and that field is NOT protected in the field protection settings, then the 008 in the incoming record should replace the existing 008
• If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should be added, but only if the data in it is different from the data in that field in the existing record. Since the incoming 006 is different, it is added, and the updated record ends up with three 006 fields.
• So the updated record would have
  • 001: in001
  • 008: 456
  • 006: abc
  • 006: def
  • 006: xyz

Example 3

• Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
• Existing record has following control fields
  • 001: in001
  • 006: abc
  • 006: def
• Overlay is performed by a record with these fields
  • 001: ybp74
  • 006: abc
  • 006: xyz
• If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should only be added if the data in it is different from the data in that field in the existing record.
• So the updated record would have the two 006s from the existing record, plus the unique 006 from the incoming record, but would not have the duplicate 006 from the incoming record (abc)
  • 001: in001
  • 006: abc
  • 006: def
  • 006: xyz

Example 4

• Field protection setting = Field = 006, Ind 1/2 = *, Subfield = *, Data = *
• Existing record has following control fields
  • 001: in001
  • 006: abc
  • 006: def
• Overlay is performed by a record with these fields
  • 001: ybp74
  • 006: abc
• If it's a repeatable field (006, 007), and the field is protected (like the 006), then the incoming field should only be added if the data in it is different from the data in that field in the existing record. In this case, the incoming 006 is the same as an existing 006
• So the updated record would have
  • 001: in001
  • 006: abc
  • 006: def

Example 5

• No field protection settings, or no Update MARC Bib action in Job Profile
• Existing record has following control fields
  • 001: in001
  • 006: abc
  • 006: def
• Overlay is performed by a record with these fields
  • 001: ybp74
  • 006: abc
  • 006: xyz
• Because there are no field protections, all fields in the SRS MARC record (except 001 and 999 ff) are replaced with the incoming MARC data
• So the updated record would have the 001 from the existing record, and the 006s from the incoming record
  • 001: in001
  • 006: abc
  • 006: xyz

Additional Juniper records that can be used for reproducing and testing:
(HRID is first, then the OCLC number)
ak00005007612 42980246
in7023981 42980246
ins00007267522 42980246
ak00005007664 43311780
ins00007267543 43311780
in7022003 43311780
in8779566 43311780
ak00005507053 36485985
ak00005003633 36485985
ak00005275045 36485985
ak00005007101 42668854
in7022076 42668854
ins00007266968 42668854

Interested parties:

charlotte abreaux kgambrell