Match and action profiles cannot be re-used in an import job profile - short term fix (Poppy)

Overview: Data import job profiles can be created using the same match profiles more than once, but once the job profile is saved the reused components are repeated under each section. For instance, Chicago needs to create a job profile that matches the 035$z of the incoming record to the OCLC number in the instance, then submatch to the instance status is equal to a constant value (e.g., Batch Loaded), and then submatch to the holdings type (e.g., Electronic). If there is not match on the 035$z to the OCLC number, then the import should match the 035$a of the incoming record to the OCLC number in the instance record with an instance submatch where the instance status is equal to a constant value and the holdings type is equal to a constant value. The submatches are the same for each of the components. This is not currently possible without creating unique match profile for each time a match is needed to be used in an import job profile.

This bug does not require importing anything, only building the various profiles and assembling the job profile

Steps to Reproduce:

1. Log into Folio Snapshot
2. Go to Settings > Data Import.
3. Create a new field mapping profile for an instance and save it
   1. Name: Update instance
   2. Cataloged date: Select Today from dropdown list
   3. Instance status: select any value from the dropdown list
   4. Administrative notes
      1. Add these to existing
      2. "Updated via OCLC match"
4. Create a new field mapping profile for a holdings and save it
   1. Name: Update holdings temp location
   2. Administrative notes
      1. Add these to existing
      2. "Updated via OCLC number match"
5. Create a new action profile with the following options:
   1. Name: Update instance via OCLC number match
   2. Action: Update
   3. FOLIO record type: Instance
   4. Link the instance field mapping profile that you just created. 
6. Create a new action profile with the following options:
   1. Name: Update holdings via OCLC number match
   2. Action: Update
   3. FOLIO record type: Holdings
   4. Link the holdings field mapping profile that you just created. 
7. Create a new match profile named 035$a to OCLC with the following options:
   1. Select MARC Bib-to-Instance under existing records
   2. Field: 035
   3. In. 1: [space]
   4. In 2: [space]
   5. Subfield: a
   6. Match criterion: Exactly matches
   7. Existing instance record field: Identifier: OCLC
   8. Save and close
8. Create a new match profile named 035$z to OCLC with the following options:
   1. Select MARC Bib-to-Instance under existing records
   2. Field: 035
   3. In. 1: [space]
   4. In 2: [space]
   5. Subfield: z
   6. Match criterion: Exactly matches
   7. Existing instance record field: Identifier: OCLC
   8. Save and close
9. Create a new match profile named instance Status Batch Loaded
   1. Change MARC Bibliographic to Static value (submatch only)
   2. Select Instance under existing records
   3. Incoming Static value (submatch only) record - Text: Batch Loaded
   4. Match criterion: Exactly matches
   5. Existing instance record field: Admin data: Instance status term
   6. Save and close
10. Create a new match profile named Holdings type electronic
    1. Change MARC Bibliographic to Static value (submatch only)
    2. Select holdings under existing records.
    3. Incoming Static value (submatch only) record - Text: Electronic
    4. match criterion: Exactly matches
    5. Existing holdings record field: Admin data: holdings type
    6. Save and close
11. Create a new job import profile with any name and accepted date type = MARC Bibliographic. (See screenshot in edit mode for what the profile should look like before saving.)
12. Click the plus sign and add a new match: 035$z to OCLC. 
    1. Under For matches add a new match profile: Instance Status Batch Loaded
    2. Under For matches add a new match profile: Holdings type Electronic.
    3. Under For matches add a new action profile: [the update instance profile created]
    4. Under For matches add a new action profiles: [the update instance and update holdings profiles created].
13. Under the third For non-matches heading, click the plus sign and add a new match profile: 035$a to OCLC then add the same sub-match profiles and actions as under the 035$z to OCLC match above.
    1. Under For matches add a new match profile: Instance Status Batch Loaded
    2. Under For matches add a new match profile: Holdings type Electronic.
    3. Under For matches add a new action profile: [the update instance profile created]
    4. Under For matches add a new action profiles: [the update instance and update holdings profiles created].
14. review the screen to see that match profile holdings type is present only once under each OCLC match and the update instance and update holdings actions are listed only once under each OCLC match. 
15. Save the job import profile.

 Expected Results: I expect to see a job profile that will match the 035$z to Instance OCLC number, instance status, and holdings type then update the instance and and the holdings if it is a match. If there is no match I expect to see a match on the 035$a to the instance OCLC number, then submatches to the instance status and holdings type with update actions taken if there is a match. I expect to be able to build complex matches that re-use match profiles within a single job import profile.

Actual Results: Each primary match 035$z to OCLC and 035$a to OCLC  each contain two matches to holdings type and each holding type electronic match contains two update instance and two update holdings actions.

Additional Information: This behavior is present in Nolana and FOLIO snapshot.

Interested parties: University of Chicago

BE Notes:

1. Existing workaround - create job profile step by step - save one branch (building block) with links set up, and then edit it again adding other branch using the same building block. It works, but required steps that user has to make are ugly
2. Short-term, easy to implement and safe to back-port workaround in the code - add constraint (or BE validation) when creating associations to ignore creation of duplicate association in scope of one job profile. It will allow creating profiles reusing the same building blocks (match and action profile structure). Such fix will be incomplete though - described case will be covered, but in case different building blocks would be required using same associations, we still might run into a problem.{} This solution will be implemented in scope of the current ticket.
3. Adequate long-term solution - association entity currently stores parent and detail profile ids, to ensure the association is unique, instead of actual profiles ids we need to store ids of profile wrappers (new entity) in parent and detail columns. Add table for profile wrappers - wrapper id, profile type, profile id. Changes are needed in operations over associations themselves and in functionality constructing the profile, and migration. See stories MODDICONV-310, MODDICONV-311. It is the most preferable way to go, but it will take something like 13 SP and will be too risky for back-port (will be done in Poppy). 

ORCHID Critical service patch details

1. Describe issue impact on business: Work done in Nolana to stabilize job profiles inadvertently decreased the ability to use the same matches and same actions multiple times in the same job profile. This short-term fix allows for the same matches and actions to be reused in a job profile as long as they are applied in the same relationship to each other in all sections of the profile. Longer-term, the intent is to adjust the job profiles so that matches and actions can be applied without being in parallel relationships throughout the job profile. However, per the development thoughts above, that long-term solution is significantly more work and is too risky to backport.
2. What institutions are affected? (field “Affected Institutions” in Jira to be populated): Any that use complex, branching job profiles
3. What is the workaround if exists?: None that are simple. One used in the past is to create multiple duplicative match or action profiles with the same details, but different names, and then refrain from using the same match or action profile more than once in a job profile. However that involves significant work for the DI users, especially when profiles are updated, and duplicative match/action profiles must be reviewed to ensure they are kept synchronized. It also introduces possibility of error, if the duplicative profiles are not kept synchronized in terms of any manual edits.
4. What areas will be impacted by fix (i.e. what areas need to be retested): Data import profile creation and import with the 3 TestRails associated with this Jira. Plus testing by SMEs at Cornell and Chicago to review/confirm.
5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high). The work is complete and consumed 1 SP of effort. Significant effort went into teasing apart a longer-term solution and documenting on separate Jiras, plus creating TestRails and manually testing. Risk is minimal.
6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO. Test cases already exist and level of effort is .5-1 day, spread across manual testers, PO, and SMEs
7. What is the roll back plan in case the fix does not work? The patch can be rolled back if necessary, and re-evaluated

NOLANA Critical service patch details

1. Describe issue impact on business: Work done in Nolana to stabilize job profiles inadvertently decreased the ability to use the same matches and same actions multiple times in the same job profile. This short-term fix allows for the same matches and actions to be reused in a job profile as long as they are applied in the same relationship to each other in all sections of the profile. Longer-term, the intent is to adjust the job profiles so that matches and actions can be applied without being in parallel relationships throughout the job profile. However, per the development thoughts above, that long-term solution is significantly more work and is too risky to backport.
2. What institutions are affected? (field “Affected Institutions” in Jira to be populated): Any that use complex, branching job profiles
3. What is the workaround if exists?: None that are simple. One used in the past is to create multiple duplicative match or action profiles with the same details, but different names, and then refrain from using the same match or action profile more than once in a job profile. However that involves significant work for the DI users, especially when profiles are updated, and duplicative match/action profiles must be reviewed to ensure they are kept synchronized. It also introduces possibility of error, if the duplicative profiles are not kept synchronized in terms of any manual edits.
4. What areas will be impacted by fix (i.e. what areas need to be retested): Data import profile creation and import with the 3 TestRails associated with this Jira. Plus testing by SMEs at Cornell and Chicago to review/confirm.
5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high). The work is complete and consumed 1 SP of effort. Significant effort went into teasing apart a longer-term solution and documenting on separate Jiras, plus creating TestRails and manually testing. Risk is minimal.
6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO. Test cases already exist and level of effort is .5-1 day, spread across manual testers, PO, and SMEs
7. What is the roll back plan in case the fix does not work? The patch can be rolled back if necessary, and re-evaluated