Details
-
New Feature
-
Status: Open (View Workflow)
-
TBD
-
Resolution: Unresolved
-
None
-
None
-
None
-
-
DevOps: Sprint 95
-
Core: Platform
Description
The PgUtil.streamGet() may be used for some exceptionally large data sets.
The current design does not seem performant resource-wise (memory consumption, network bandwidth, and to some extend time).
This happens because the entire jsonb appears to be return from the database to the backend module before it casts/converts/maps that response to some smaller object.
Ideally, provide a way to:
- Get only a selection of some fields, such as hrid.
- Get that response without object mapping.
- Maintain stream support.
- May involve improving *.raml functionality to designate non-object results.
- Consider additional actions on fields, such as select encode((jsonb->>'id')::bytea, 'base64') from instance;.
- Potentially get the table columns and not just jsonb keys/properties (get 'id' instead of 'jsonb->id') for performance reasons.
This potentially blocks MODINVSTORE-439, but can alternatively be considered tech-debt to instead not block and utilize the currently functional but less performant implementation of PgUtil.streamGet().
TestRail: Results
Attachments
Issue Links
- relates to
-
RMB-554 PgUtil.getAllIds
-
- Open
-
-
MODINVSTOR-439 Create bulk download endpoint to save id UUIDs
-
- Closed
-
-
MODINVSTOR-452 Create size optimized bulk download endpoint to save id UUIDs
-
- Open
-
-
MODINVSTOR-465 Bulk download Instance UUIDs OOM
-
- Closed
-