NFR: Increase security of Kafka for Remote storage

Current situation or problem: Remote storage its transactions are direct Kafka connections.

There were some concerns raised in the community regarding how secure the direct connection will be. To address these concerns, the new solution was designed: https://wiki.folio.org/display/DD/Temporary+Kafka+security+solution.
The solution was reviewed and approved by the Security group and Tech Council.

Multi-tenancy on Kafka's side is implemented for the modules differently, so it will take time to make the changes in them that unify the multi-tenancy approach.
However, the direct Kafka connections should be secured in R1, so a simplified version of the solution is proposed for now.

In scope 

• Add module-level Kafka user credentials support to Remote storage. The credentials should be provided to all producers and consumers of a module with other Kafka client settings. Changes in PubSub are required since once Kafka authentication and authorization are enabled, the PubSub will need to pass through them as well.
• Add TLS support to the same modules.
  Same here, the settings should be provided to all producers and consumers of a module with other Kafka client settings.
•  

How it could be implemented:

• ModuleDescriptor should be updated to include the new Kafka settings: TLS, and, for now, user credentials (the credentials later could be injected to container a different way, for instance, as EnvironmentVariables)
• Update a class that represents Kafka config
• Update a class(es) that creates and assigns the config to Kafka producers and consumers
• Test the updates

Out of scope
This work is also needed for ElasticSearch and Data import, but those applications/modules are managed by other dev teams

Proposed solution/How it could be implemented:

• ModuleDescriptor should be updated to include the new Kafka settings: TLS, and, for now, user credentials (the credentials later could be injected to container a different way, for instance, as EnvironmentVariables)
• Update a class that represents Kafka config
• Update a class(es) that creates and assigns the config to Kafka producers and consumers
• Test the updates

Links to additional info
https://wiki.folio.org/display/DD/Temporary+Kafka+security+solution.

Questions