Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-2592

ElasticSearch - APIs for search and data import

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Closed (View Workflow)
    • Priority: TBD
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Template:
    • Epic Link:
    • Development Team:
      ไป่€…ๆ— ๆ•Œ "Benevolence"
    • Calculated Total Rank:
      11
    • Rank: Cornell (Full Sum 2021):
      R2
    • Rank: GBV (MVP Sum 2020):
      R2
    • Rank: MO State (MVP June 2020):
      R4

      Description

      *High level feature description:* Provide an API for search and data import.

      Search will slow down with the increasing number of data in the databases. The ES search engine will help increase search efficiency when data volume is big and data relationship is complicated.

      Provide an API for search and data import. The ES component includes three modules: mod-search-es, mod-data-loader-es and raml-module-builder. The mod-search-es provides an interface for business modules to call. The mod-search-es is the backend to convert the database data into index data. The raml-module-builder will monitor the changes in the database and send data to be indexed to the mod-data-loader-es.

      Each individual business module decides whether to introduce the ES and where to introduce it. In order to connect to the ES modules, 1.Database query code should be changed to call the mod-search-es; 2. Integrate the raml-module-builder with the mod-inventory-storage.

      *Detailed plans:*

      1. mod-search-es
      1) This module is used for checking ES data.
      2) Configure some configuration information of import ES data, such as database connection configuration, database driver, database user name, database password, SQL statement of database query, configuration of source data fields and target fields, as well as the configuration content of converter used.

      2. mod-data-loader-es
      1) This module is used to parse the read configuration information and import the data into ES.
      2) Listen to the message content in Kafka and parse the data into ES after transformation.

      3. raml-module-builder
      Modify THE RMB module to listen for HTTP requests under the configured route, forward the requested content to Kafka, and then synchronize the data to ES through MOd-data-Loader-ES.

        TestRail: Results

          Attachments

            Activity

              People

              Assignee:
              Lucy Tiewei Liu
              Reporter:
              cboerema Cate Boerema
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  TestRail: Runs

                    TestRail: Cases