SPIKE: propose prevention of DoS via CQL query

Description

Several application endpoints accessible by normal users perform CQL (Common Query Language) queries leveraging CQL-Java. CQL is a formal language for representing queries to information retrieval systems such as web indexes, bibliographic catalogs and museum collection information.1 The application can be forced to perform an OR expression over objects that match a certain patterns. As a result, the CQL engine is forced to do very expensive queries to search and parse the result of the operations. Although it appears that the full result of the operation is filtered downstream when returned to the user, It was noticed that each request to the target endpoint would take around 29 seconds to respond back. An attacker can exploit this by performing a denial of service attack by requesting several GET requests simultaneously or in short intervals, resulting in the exhaustion of the server’s memory resources.

Steps to reproduce

Perform the following request with the thread count set to 20 and throttle time set to 1 milliseconds. Substitute the value of X-Okapi-Token header with a token belonging to a user that has inventory.instances.collection.get permission.

GET /inventory/instances?limit=1000&query=%28keyword%20all%20%22ncc%22%29%20or%201=1
urldecoded query: (keyword all "ncc") or 1=1
Users can use the Inventory front-end "Query search" slot to enter any CQL: https://folio-snapshot.aws.indexdata.com/inventory?qindex=querySearch&sort=Title

Observe that all inventory related endpoints will become unresponsive

Acceptance criteria

Propose an approach that will prevent heavy queries to cause a module to become unresponsive. Provide a PoC implementation for review.

Potential solutions:

Query validation based:

• RMB: validate query length
• RMB: validate query structure (max number of OR expression)
• RMB: validate if query field and relation is backed up by an index (in schema.json, reject if not)
• RMB: validate known problematic searches (e.g cql.allRecords)
• RMB: wide truncation queries

Query execution based:

• Can Postgres analyze query runtime?
• Can Postgres limit the runtime for a SELECT
• If not, can we timeout a connection for long running queries