Uploaded image for project: 'RAML Module Builder'
  1. RAML Module Builder
  2. RMB-657

Default 60s connectionReleaseDelay prevents streaming large data

    XMLWordPrintable

Details

    • Standard Bug Write-Up Format
    • CP: sprint 91, CP: sprint 92
    • 0.5
    • Core: Platform
    • Q2 2020

    Description

      Overview:
      A database connection is cancelled when it takes more than 60 seconds before the first record is sent or if a pause of 60 seconds between two record.

      While testing bulk instance id download, noticed that mysterious 500 closed error. It turns out that the default 60s connectionReleaseDelay parameter kicked in. While the parameter can be adjusted when starting mod-inventory-storage module, it would be nice if the connection used by streaming operation is not treated as idle.

      Steps to Reproduce:
      This problem was observed in bugfest-golden env, but you can easily reproduce it by doing

      1. Use a larger dataset. For example, nightly perf data set.
      2. Start mod-inventory-storage with a smaller connectionReleaseDelay value like 10s
      3. Make API call to /instance-bulk/ids?limit=2147483647&query=(keyword="") sortby title

      Expected Results:
      API call finishes OK

      Actual Results:

      {
        "ids": [],
        "totalRecords": 2760630,
        "resultInfo": {
          "totalRecords": 2760630,
          "facets": [],
          "diagnostics": [
            {
              "code": "500",
              "message": "closed"
            }
          ]
        }
      }
      

      Database has log entry about broken pipe

      2020-06-26 21:24:53.376 UTC [720] LOG:  could not send data to client: Broken pipe
      2020-06-26 21:24:53.376 UTC [720] STATEMENT:  SELECT jsonb,id FROM supertenant_mod_inventory_storage.instance WHERE true ORDER BY left(lower(f_unaccent(instance.jsonb->>'title')),600), lower(f_unaccent(instance.jsonb->>'title'))
      2020-06-26 21:24:53.376 UTC [720] FATAL:  connection to client lost
      2020-06-26 21:24:53.376 UTC [720] STATEMENT:  SELECT jsonb,id FROM supertenant_mod_inventory_storage.instance WHERE true ORDER BY left(lower(f_unaccent(instance.jsonb->>'title')),600), lower(f_unaccent(instance.jsonb->>'title'))
      

      Cause
      RMB incorrectly implements connectionReleaseDelay by cancelling connections that are in use.

      Solution
      Disable the current wrong implementation of connectionReleaseDelay.
      Consequently connections that are no longer in use will stay in the connection pool forever (but can be re-used).
      See RMB-663 for the correct implementation.

      Note
      This bug does not affect the long running database tasks when upgrading a module to a new version.

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                julianladisch Julian Ladisch
                hji Hongwei Ji
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases