Details
-
New Feature
-
Status: Closed (View Workflow)
-
P3
-
Resolution: Done
-
None
-
None
-
-
Large < 10 days
-
XL < 15 days
-
Low
-
XXL < 30 days
-
-
Folijet
-
-
R2
-
R1
-
R1
-
R1
-
R1
-
R1
-
R1
-
R1
Description
Help the user set rules and profiles for dealing with incoming batch loads:
- what type of incoming file
- from what source
- what FOLIO apps/record types should be affected
- how to parse/map incoming data
- create profiles related to the parsing & matching rules, including defaults
- be able to mix and match pieces of profiles (or subprofiles) to create overall profiles - with varying mapping rules, matching rules, defaults, etc
- matching rules and hierarchies, and what happens (create new, overlay, merge, delete or mark for deletion)
- package up the rules (profile) and data to deliver to the appropriate FOLIO loader apps, to CRUD the proper kinds of records
Key elements:
- Apps
- ui-data-import (only landing page in this epic; other pages will come in subsequent epics)
- mod-data-import, including file upload component
- mod-source-record-storage
- mod-source-record-manager
- File upload component
- UI:
- App landing page, with preview and running jobs, most recent 25 imports, file upload area
- Outline of settings pages for file extensions, job profiles, match profiles, action profiles, data mapping profiles
- File upload component
See https://docs.google.com/drawings/d/1IN2t0T838HtHytNctdugQN0nL8_KSJ-PzysQxSyZ1Hc/edit for a schematic
TestRail: Results
Attachments
Issue Links
- defines
-
UXPROD-47 Batch Importer (Bib/Acq)
-
- Analysis Complete
-
- is continued by
-
UXPROD-1656 Data Import (Batch Importer for Bib Acq) Q1 2019 Technical and NFR work
-
- Closed
-
- relates to
-
MODDATAIMP-1 Initial design for Batch Loader flow
-
- Closed
-
-
MODDATAIMP-2 Batch Loader: Create Folio app skeleton
-
- Closed
-
-
MODDATAIMP-3 SPIKE: PoC - usage of org.marc4j library
-
- Closed
-
-
MODDATAIMP-4 Spike: Figure out relationship of Folijet batch loader and CoreTeam mod-data-loader
-
- Closed
-
-
MODDATAIMP-5 SPIKE: Batch Loader Overview
-
- Closed
-
-
MODDATAIMP-6 UI: Search panel
-
- Closed
-
-
MODDATAIMP-10 SPIKE: UI: Upload component
-
- Closed
-
-
MODDATAIMP-11 UI: Drag&Drop
-
- Closed
-
-
MODDATAIMP-12 UI: Data import snapshot result
-
- Closed
-
-
MODDATAIMP-13 UI: Results page
-
- Closed
-
-
MODDATAIMP-14 Create mod-data-import module
-
- Closed
-
-
MODDATAIMP-16 SPIKE: Investigate possibility to call one service in several modules through OKAPI
-
- Closed
-
-
MODDATAIMP-19 Create sample module using vertx-rx-java
-
- Closed
-
-
MODDATAIMP-22 SPIKE: Design Backend functionality to support "File Upload"
-
- Closed
-
-
MODDATAIMP-24 UI: Drag&Drop: nest into panel
-
- Closed
-
-
MODDATAIMP-25 UI: Drag&Drop: Configuration
-
- Closed
-
-
MODDATAIMP-26 UI: Drag&Drop: Markup
-
- Closed
-
-
MODDATAIMP-27 Create activify diagrams for 4 usecases with Matching and Action profiles
-
- Closed
-
-
MODDATAIMP-28 Add CRUD API for file upload and a upload definition
-
- Closed
-
-
MODDATAIMP-29 Add services to to store and get uploaded files
-
- Closed
-
-
MODDATAIMP-31 Use description fields in RAML JSON schemas
-
- Closed
-
-
MODDATAIMP-32 Refine FOLIO-Data-Import-HighLevel-Arch-Outline
-
- Closed
-
-
MODDATAIMP-33 Create service's skeleton for upload functionality
-
- Closed
-
-
MODDATAIMP-34 Add implementation to UploadDefintion's methods
-
- Closed
-
-
MODDATAIMP-35 Add implementation to File Upload\Delete methods
-
- Closed
-
-
MODDATAIMP-36 Automate new modules’ registration within folio/testing-backend Vagrant box.
-
- Closed
-
-
MODDATAIMP-37 Security vulnerability reported in jackson-databind
-
- Closed
-
-
MODDATAIMP-38 Create sample data to return JobExecution and Log entities
-
- Closed
-
-
MODDATAIMP-39 Create service for file chunking
-
- Closed
-
-
MODDATAIMP-41 Add implementation to create job execution call in UploadDefinition Service
-
- Closed
-
-
MODDATAIMP-42 Create sample data for the JobExecution entity
-
- Closed
-
-
MODDATAIMP-43 Create sample data for the Log entity
-
- Closed
-
-
MODDATAIMP-44 Create common RAML schema storage git
-
- Closed
-
-
MODDATAIMP-45 Implement MARC file reader for chunking service
-
- Closed
-
-
MODDATAIMP-46 Change module configuration to use Log4J 2 logger implementation.
-
- Closed
-
-
MODDATAIMP-47 Make a description of current work with transactions and add new approach
-
- Closed
-
-
MODDATAIMP-48 SPIKE: Create BackEnd Documentation for File Upload component
-
- Closed
-
-
MODDATAIMP-49 Prepare module for the release
-
- Closed
-
-
MODDATAIMP-50 Add DELETE endpoint fo Upload Definition entity
-
- Closed
-
-
MODDATAIMP-51 SPIKE: test our upload implementation for the max file size it can process
-
- Closed
-
-
MODDATAIMP-52 Create Profile entity
-
- Closed
-
-
MODDATAIMP-53 Create release branches for modules owned by Folijet team.
-
- Closed
-
-
MODDATAIMP-54 Add "uploadedDate" to the file model
-
- Closed
-
-
MODDATAIMP-55 422 Error during POST /change-manager/jobExecutions
-
- Closed
-
-
MODDATAIMP-56 Delete file from storage on DELETE file by id
-
- Closed
-
-
MODDATAIMP-57 Create shared library with data utils for data-import modules
-
- Closed
-
-
MODDATAIMP-58 Create shared library with data utils for data-import modules
-
- Closed
-
-
MODDATAIMP-64 Fix security vulnerabilities reported in jackson-databind >= 2.9.0, < 2.9.8
-
- Closed
-
-
MODDATAIMP-66 SPIKE: Become familiar with Apache Kafka® - a distributed streaming platform.
-
- Closed
-
-
MODDATAIMP-88 Data Import architecture review preparation
-
- Closed
-
-
MODDATAIMP-89 "Data Import" use cases analysis.
-
- Closed
-
-
MODDATAIMP-100 Prepare "Data Import" first stage demo.
-
- Closed
-
-
MODDATAIMP-101 Don't work ParallelFileChunkingProcessor
-
- Closed
-
-
MODDATAIMP-102 Update documentation for file upload API
-
- Closed
-
-
MODDATAIMP-103 JobProfileInfo don't updates for job execution
-
- Closed
-
-
MODSOURCE-1 Create mod-source-record-storage module
-
- Closed
-
-
MODSOURCE-2 Create REST endpoint to return a list of "Source Change Jobs"
-
- Closed
-
-
MODSOURCE-4 Create REST endpoint to return a list of logs for "Source Change Jobs"
-
- Closed
-
-
MODSOURCE-6 Add CRUD API for Snapshot at mod-source-record-storage
-
- Closed
-
-
MODSOURCE-7 Add CRUD API for Record entity at mod-source-record-storage
-
- Closed
-
-
MODSOURCE-8 Add GET method for Result entity at mod-source-record-storage
-
- Closed
-
-
MODSOURCE-13 Add metadata to Result entity
-
- Closed
-
-
MODSOURCE-14 Change totalRecords from recordsCollection to return overall total
-
- Closed
-
-
MODSOURCE-15 Spike: Investigate population of metadata
-
- Closed
-
-
MODSOURCE-18 Prepare module for release
-
- Closed
-
-
MODSOURCE-19 Create distributable HTTP Client for all services provided by the module mod-source-record-storage
-
- Closed
-
-
MODSOURCE-21 Fix security vulnerabilities reported in jackson-databind >= 2.8.0, < 2.8.11.3
-
- Closed
-
-
MODSOURCE-23 Create endpoint
-
- Closed
-
-
MODSOURCE-24 Create stub snapshot if module deployed in test mode
-
- Closed
-
-
MODSOURCE-25 Implement endpoint
-
- Closed
-
-
MODSOURCE-26 Provide examples on how to use the endpoint
-
- Closed
-
-
MODSOURMAN-1 Create Source records Manager module
-
- Closed
-
-
MODSOURMAN-3 Create technical design for Change Manager component from mod-source-record-manager
-
- Closed
-
-
MODSOURMAN-4 Create a RAML definition and endpoints with stub implementation for "Metadata Service" and "Metadata Provider"
-
- Closed
-
-
MODSOURMAN-5 Make a design for creation and registration of a new loader job
-
- Closed
-
-
MODSOURMAN-6 Make a design for receiving and processing source data
-
- Closed
-
-
MODSOURMAN-7 MetadataRepository: Implement API for JobExecution entities
-
- Closed
-
-
MODSOURMAN-8 Extend JobExecution entity with a new fields
-
- Closed
-
-
MODSOURMAN-9 Create ChangeManager component, expose API for JobExecutions creation
-
- Closed
-
-
MODSOURMAN-11 Create ChangeEngine component for parsing raw records
-
- Closed
-
-
MODSOURMAN-18 Create endpoint for ChangeManager to update JobExecution
-
- Closed
-
-
MODSOURMAN-19 ChangeManager: API for receiving chunks of raw (source) data with a stub implementation
-
- Closed
-
-
MODSOURMAN-23 Implement methods for MetadataRepository
-
- Closed
-
-
MODSOURMAN-24 Prepopulate repository with sample JobExecutions data
-
- Closed
-
-
MODSOURMAN-25 Change module configuration to use Log4J 2 logger implementation.
-
- Closed
-
-
MODSOURMAN-26 Create SQL script to prepopulate JobExecutions in AWS env.
-
- Closed
-
-
MODSOURMAN-27 Change module structure to the common modules structure
-
- Closed
-
-
MODSOURMAN-28 Create RAML description for the new REST endpoint
-
- Closed
-
-
MODSOURMAN-29 REST API implementation
-
- Closed
-
-
MODSOURMAN-30 "Log" should be a DTO not a model object
-
- Closed
-
-
MODSOURMAN-31 Change data type from "String" to "Date-Time" for all date-time fields in all schemas for DataImport App
-
- Closed
-
-
MODSOURMAN-32 Add a new field "UI Status" to the jobExecution model entity and related DTOs (jobExecutionDto, logDto)
-
- Closed
-
-
MODSOURMAN-34 Prepare module for the release
-
- Closed
-
-
MODSOURMAN-35 Apply backend sorting for the Log entities
-
- Closed
-
-
MODSOURMAN-36 Select only COMMITTED JobExecutions (Logs) on backend
-
- Closed
-
-
MODSOURMAN-37 Make request param "landingPage" optional
-
- Closed
-
-
MODSOURMAN-38 Not enough permissions to call SnapshotService
-
- Closed
-
-
MODSOURMAN-40 Request to GET Jobs is failed
-
- Closed
-
-
MODSOURMAN-41 Add endpoints for updating JobExecution status and getting JobExecution children by id
-
- Closed
-
-
MODSOURMAN-43 Add endpoint for setting JobProfileName and Simplify logic of Update method for JobExecution
-
- Closed
-
-
MODSOURMAN-44 Optimize code in source-record-manager
-
- Closed
-
-
MODSOURMAN-45 Eliminate potential problems with JobExecution status update and query limit
-
- Closed
-
-
MODSOURMAN-46 Validate status on JobExecution status update
-
- Closed
-
-
MODSOURMAN-47 Exclude DISCARDED JobExecutions from JobExecutionCollectionDto
-
- Closed
-
-
MODSOURMAN-48 Override default limit value of returned JobExecution children
-
- Closed
-
-
MODSOURMAN-101 Data Import online API road map (without using file upload functionality)
-
- Closed
-
-
UIDATIMP-4 Create landing page for Data Import app - DUPLICATE!
-
- Closed
-
-
UIDATIMP-5 Create data import settings page 1: page, 1st, and 2nd panes
-
- Closed
-
-
UIDATIMP-44 Spike: Investigate the way to implement the card view for profiles
-
- Closed
-
-
UIDATIMP-49 Spike: Investigate state management on UI
-
- Closed
-
-
UIDATIMP-52 Change contracts
-
- Closed
-
-
UIDATIMP-70 Release v1.0.0 for Q4 2018
-
- Closed
-
-
UIDATIMP-71 Hide popover when user clicks on the link button
-
- Closed
-
-
UIDATIMP-74 Add document icon to upload file items
-
- Closed
-
-
UIDATIMP-75 Setup BigTest environment and write tests for first pane on the landing page
-
- Closed
-
-
UIDATIMP-78 Mock backend for testing
-
- Closed
-
-
UIDATIMP-81 Translation problem on Data Import landing page
-
- Closed
-
-
UIDATIMP-82 Change jobExecutions status to much uiStatus from the backend
-
- Closed
-
-
UXPROD-130 Parse MARC21 format data records
-
- Closed
-
-
UXPROD-1376 CRUD for File Extensions
-
- Closed
-
-
UXPROD-1377 CRUD for Job Profiles
-
- Closed
-
-
UXPROD-1378 CRUD for Match Profiles
-
- Closed
-
-
UXPROD-1379 CRUD for Action Profiles
-
- Closed
-
-
UXPROD-1380 CRUD for MARC Field Mapping Profiles, part 1
-
- Closed
-
-
UXPROD-1381 CRUD for EDIFACT Invoice Field Mapping Profiles
-
- Closed
-
-
UXPROD-1382 CRUD for Delimited Data Mapping Profiles
-
- Draft
-
-
UXPROD-1405 Data Import (Batch Importer for Bib Acq) - Misc navigation & screen issues
-
- Draft
-
-
UXPROD-2074 Bring Action Profiles to life: prep work
-
- Closed
-
-
UXPROD-2075 MARC Bib Field Mapping Profiles Details for Inventory records
-
- Closed
-
-
UXPROD-2209 Bring Action Profiles to life: next work
-
- Closed
-
-
UIDATIMP-1 Batch Loader: Create Stripes app
-
- Closed
-
-
UIDATIMP-2 Batch loader initial stripes app structure
-
- Closed
-
-
UIDATIMP-3 Setup Jenkins CI loop for ui-data-import module
-
- Closed
-
-
UIDATIMP-19 Data Import landing page Scenario 1: App Icon
-
- Closed
-
-
UIDATIMP-20 Data Import landing page Scenario 2: Basic Page Layout
-
- Closed
-
-
UIDATIMP-21 Data Import landing page Scenario 3: Menu Settings Button
-
- Closed
-
-
UIDATIMP-22 Data Import landing page Scenario 4: First Pane Layout
-
- Closed
-
-
UIDATIMP-23 Data Import landing page Scenario 5: First Pane Top Half Layout
-
- Closed
-
-
UIDATIMP-24 Data Import landing page Scenario 6: First Pane "Preview Ready" Box
-
- Closed
-
-
UIDATIMP-25 Data Import landing page Scenario 7: First Pane "Preview is Running" Box
-
- Closed
-
-
UIDATIMP-26 Data Import landing page Scenario 8: Change from Preview Running to Ready Box
-
- Closed
-
-
UIDATIMP-27 Data Import landing page Scenario 9: First Pane Bottom Half Layout
-
- Closed
-
-
UIDATIMP-28 Data Import landing page Scenario 10: First Pane Running Jobs Box
-
- Closed
-
-
UIDATIMP-29 Data Import landing page Scenario 11: Second Pane Layout
-
- Closed
-
-
UIDATIMP-31 Create a reusable stripes component for "Preview" & "Running Jobs" boxes
-
- Closed
-
-
UIDATIMP-32 Gather UI related questions
-
- Closed
-
-
UIDATIMP-33 Create UI for File Upload, Part 1: Invoking the File Upload component
-
- Closed
-
-
UIDATIMP-35 Create UI for File Upload, Part 2: Styling for "In Process" File Upload Pane
-
- Closed
-
-
UIDATIMP-37 Create UI for File Upload, Part 3: Action & Styling for "In Process" file uploads
-
- Closed
-
-
UIDATIMP-38 Create UI for File Upload, Part 4: Styling for "Completed" file uploads
-
- Closed
-
-
UIDATIMP-39 Create UI for File Upload, Part 5: Action when file upload begins
-
- Closed
-
-
UIDATIMP-40 Create End-of-list shared component
-
- Closed
-
-
UIDATIMP-43 Data Import landing page Scenario 12: Third Pane Layout
-
- Closed
-
-
UIDATIMP-46 Create UI for File Upload, Part 4: Styling for Error: multiple file types
-
- Closed
-
-
UIDATIMP-50 Does not show proper app label in app bar when locale is changed to Arabic
-
- Closed
-
-
UIDATIMP-63 Creating UI for File Upload. When file uploading of particular file(s) has failed
-
- Closed
-
-
UIDATIMP-65 SPIKE: Create Frontend Documentation for File Upload component
-
- Closed
-
-
UIDATIMP-67 Creating UI for File Upload. User navigation when file upload is in progress
-
- Closed
-
-
UIDATIMP-72 Creating UI for File Upload. Deleting failed file from server
-
- Closed
-
-
UXPROD-1220 Ability to assign Tags to Data Import profiles
-
- Closed
-