Spike: Complete jobs that cannot update DB after failure

Purpose/Overview:

Requirements/Scope:
During the tests in bugfest environment, the database was restarted shortly after the data export started. As a result, the job was stuck in the in Running state. The log showed

Invalid token: Unexpected user response code 500 for user id 9eb67301-6f6e-468f-9b1a-6134dc39a684[\r][\n]

After this exception, the logs indicate that the job infact completed with error saying that there were no records to export because neither SRS nor inventory returned any records because of above exception. However, as the DB was not available the status in job_execution table could not be updated, there by leaving the job in IN_PROGRESS state.

Approach:

Acceptance criteria:

• come up with a way to change the status of the job if it is stuck in IN_PROGRESS state for very long
• In cases that the DB is not reachable the job should fail and appropriate error logs contains valid information.
• Ideally, the job should attempt to get the new token and continue the job (if this is possible)
• Document findings