Uploaded image for project: 'RAML Module Builder'
  1. RAML Module Builder
  2. RMB-432

fulltext: word splitting and punctuation removal

    XMLWordPrintable

Details

    • CP: sprint 69
    • 3
    • Core: Platform

    Description

      This CQL query

      title=Dell'Emulazione e dell'Influenza

      produces this SQL query:

      WHERE to_tsvector('simple', f_unaccent(instance.jsonb->>'title')) @@ to_tsquery('simple', f_unaccent('Dell'Emulazione<->e<->dell''Influenza'))

      Note that punctuation like Apostrophe and Right Single Quotation Mark (U+2019) are not removed. This cannot match because Postgres' to_tsvector removes them:

      select s, to_tsvector('simple', s) from (values('Dell'Emulazione e dell''Influenza')) as t(s);
                s                 |                  to_tsvector
----------------------------------+-----------------------------------------------
 Dell'Emulazione e dell'Influenza | 'dell':1,4 'e':3 'emulazione':2 'influenza':5

      How to fix:
      Change the word splitting and ftTerm to work in a similar way as to_tsvector:
      https://github.com/folio-org/raml-module-builder/blob/v26.2.0/cql2pgjson/src/main/java/org/folio/cql2pgjson/CQL2PgJSON.java#L671

      TestRail: Results

        Attachments

          Issue Links

            blocks

            Bug - A problem which impairs or prevents the functions of the product. RMB-438 contributor search returns no results when name contains "-"

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. UIIN-594 Title search: Apostrophe/Right Single Quotation Mark (U+2019) not found

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. UIIN-676 Search on titles with apostrophe fails

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            duplicates

            Task - A task that needs to be done. RMB-407 Move ftTerm to Cql2SqlUtil and unit test it

            • TBD - To be determined
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. RMB-415 queryByLike should warn if caseSensitive / removeAccents to do not match

            • TBD - To be determined
            • Closed
            relates to

            Bug - A problem which impairs or prevents the functions of the product. RMB-537 f_unaccent single quote fullText tsquery sql injection

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. CIRCSTORE-138 Filtering Requests by Request Status doesn't return any rows

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. RMB-439 Remove "-" token for full search to unblock CIRCSTORE-138

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. RMB-455 do not apply lower for fulltext

            • TBD - To be determined
            • Closed

            Activity

              People

                julianladisch Julian Ladisch
                julianladisch Julian Ladisch
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases