Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-2929

NFR: Increase security of Kafka for Data Import and PubSub



    • Very Small (VS) < 1day
    • Medium
    • Large < 10 days
    • Folijet
    • 113
    • R2 2021


      Next step: abreaux plan a meeting the week of 19 April with team leads and POs and Vasily to discuss implementation, and draft spikes and stories

      Latest documentation: https://wiki.folio.org/pages/viewpage.action?spaceKey=FOLIJET&title=Enabling+SSL+and+ACL+for+Kafka

      Current situation or problem: In Iris, Data import has migrated most of its transactions (but not all) to direct Kafka connections instead of mod-pubsub.

      There were some concerns raised in the community regarding how secure the direct connection will be. To address these concerns, the new solution was designed: https://wiki.folio.org/display/DD/Temporary+Kafka+security+solution.
      The solution was reviewed and approved by the Security group and Tech Council.

      Multi-tenancy on Kafka's side is implemented for the modules differently, so it will take time to make the changes in them that unify the multi-tenancy approach.
      However, the direct Kafka connections should be secured in R1, so a simplified version of the solution is proposed for now.

      In scope

      • Add module-level Kafka user credentials support to Data import and PubSub modules. The credentials should be provided to all producers and consumers of a module with other Kafka client settings. Changes in PubSub are required since once Kafka authentication and authorization are enabled, the PubSub will need to pass through them as well.
      • Add TLS (Transport Layer Security) support to the same modules. Same here, the settings should be provided to all producers and consumers of a module with other Kafka client settings.

      Out of scope
      This work is also needed for ElasticSearch and Remote Storage, but those applications/modules are managed by other dev teams

      Proposed solution/How it could be implemented:

      • ModuleDescriptor should be updated to include the new Kafka settings: TLS, and, for now, user credentials (the credentials later could be injected to container a different way, for instance, as EnvironmentVariables)
      • Update a class that represents Kafka config
      • Update a class(es) that creates and assigns the config to Kafka producers and consumers
      • Test the updates

      Links to additional info


      TestRail: Results


          Issue Links



                abreaux Ann-Marie Breaux
                abreaux Ann-Marie Breaux
                Ann-Marie Breaux Ann-Marie Breaux
                Oleksii Kuzminov Oleksii Kuzminov
                0 Vote for this issue
                1 Start watching this issue



                  TestRail: Runs

                    TestRail: Cases