Rancher Improvement: Implement Data Migration Functionality

Current situation or problem:

Currently there is no possibility to run migration on performance rancher. That's why we can't test migration from old release version to new one before bug fest. It is not optimal situation because teams need to have possibility test data migration (including  data migration performance) during development to identify and solve possible issues on early stage. 

Probably this functionality also can be used to the schema migration testing. Which help  us eliminate a lot of efforts from the development teams.

(currently schema migration testing is done by the teams locally using vagrant box and a set of data generated by a team themselves)

This functionality should allow us to identify the problematic modules from the data migration performance point of view so SA can focus on improving them and this improvement can be measured after it is implemented

In scope

• Jenkins job need to be implemented to run migration from the specified release version to the specified release version on the rancher performance environment on the specified data set.
• Modules versions need to be taken from platform-complete by  specified tag or from master branch. It should be possibility to override modules version/branch if needed (from configuration file. version/branch from the file will be finally deployed)
• Logging should be added to log time took for data migration per module

• • (added by Raman Auramau) Logged duration should be aggregated in report and available via Jenkins job
  • (added by Raman Auramau) Notification is required if upgrade job executes too long (longer than some threshold)
• We need to be sure that we are installing initial release version with compatible data set. 
• We can want to use Cornell data set (but sensitive data need to be disclosed) . We need to talk to Cornell before.  But implementation should not be blocked by this. We can start implementation on Bugfest snapshot. 
• We want to be able to update not all modules but only some specified in the list. 
• We want to be able to disable functionality to check modules compatibility so we can test upgrade from any version to any version in any time.
• We need to be able to restore the database to some state if something happened (ability to restore the database).

Out of scope
implementing this functionality for the development teams rancher envs

Preparing Cornell data for using with this env 

Use case(s)

Proposed solution/stories

Links to additional info

Questions

• Should we implement this functionality for the PTF env?
  Kitfox and PTF should discuss this

• Should Perf env should be supported for multiple teams in the same time?
  Currently it supports 3 teams at a time