# StringTemplates This document outlines the inner workings of the feature to add string templating to mod-agreements in ERM ## StringTemplate Domain Class The main thrust of this feature is a new domain class, `StringTemplate`. It looks like this: ``` class StringTemplate implements MultiTenant { String id String name String rule @Defaults(['urlProxier', 'urlCustomiser']) RefdataValue context static hasMany = [idScopes: String] // Method to output customised string based on rule/binding String customiseString(Map binding) } ``` ### Rule This domain class will store a handlebars based `rule` string, such as ``` "http://sub-hh{{replace (replace (removeProtocol inputUrl) "ebooks" "") "input.cfm" ""}}" ``` and has a method to take in a Map of variables and hand them to this rule to produce an output. ### Context Certain StringTemplates will need to be applied to certain ids, and occasionally a customised string will need to be customised again by a second StringTemplate. In order to keep track of this, we have a refdata `context` property. In our example there are two contexts, `urlProxier` and `urlCustomiser`. It is the StringTemplatingService's job to know the specific behaviour each of these needs to have. The `context` will also govern how the `idScopes` list is utilised. ### idScopes This is essentially just a list of ids, used to work out whether a given StringTemplate should apply or not. Some contexts may choose to use this as a allowlist, such as `urlCustomiser`, and some may choose to use it as a denylist, such as `urlProxier`. ## TemplatedUrl Domain Class This work also contains another domain class, `TemplatedUrl`, which exists in a list on `ErmResource`. This domain class has the shape: ``` class TemplatedUrl implements MultiTenant { String id String name String url ErmResource resource } ``` This class exists to act as a "cache" for the urls once they have been templated, allowing us to export them/display them exactly as with other fields. ## Service To go along with the new `StringTemplate` class is a service `StringTemplatingService`. This will house the business logic which determines the order/heirachy/behaviour of each StringTemplate, depending on its `context` and `idScopes`. It will also do the actual templating work. ### findStringTemplatesForId One of the methods offered by this service will take in a `String id` and return a list of StringTemplates relevant to that id. This is the method which houses the denylist/allowlist distinction above, and will return a Map of StringTemplates relevant to the given id, sorted by context: ``` [ "urlProxiers": [ [ "id": "27eefb17-5f0e-4cc7-98e1-d95b3f9825c5", "rule": "whatever-{{platformLocalCode}}", "context": [ "id": "ff8081817550a4d4017550a52efa000f" ], "name": "proxy2", "idScopes": [] ] ], "urlCustomisers": [ [ "id": "31f110e5-6faf-4502-9555-7be33b2d7fd9", "rule": "http://customise-me:{{replace inputUrl \"a\" \"b\"}}", "context": [ "id": "ff8081817550a4d4017550a52f010010" ], "name": "customiser1", "idScopes": [ "f311d130-8024-47c4-8a86-58f817dbefde", "385c06ff-2636-4f2d-8677-d28989463e75" ] ] ] ] ``` This particular response was for the id `f311d130-8024-47c4-8a86-58f817dbefde`. Notice how `proxy2` does _not_ contain this id in its `idScopes` list, and `customiser1` _does_. This is an example of the denylist/allowlist at play. ### performStringTemplates Pretty much all the other work this service does relies on actually performing these string templating operations. To that end we provide a method: `performStringTemplates`, which takes in a `String id` and a `Map binding`. This `binding` should be of the form: ``` [ inputUrl: "http://ebooks.ciando.com/book/index.cfm?bok_id=27955222", platformLocalCode: "ciando", ... ] ``` and contain all variables that need to be accessible to the template engine, the main two being inputUrl and platformLocalCode. Right now these are the only inputs, but in theory this binding could take any named variable. Firstly the method calls `findStringTemplatesForId` to return a Map as [above](#findstringtemplatesforid). Then it performs all the `urlCustomisations`, followed by all the `urlProxiers` on the initial binding, adding those to an output `Map`. Finally, we have a requirement to perform the proxies on all customised urls as well. So we run through the proximisation again for each customised url. This is an example of the Service containing the business logic dictating the order/heirachy of these operations. If the output requirement changed, or was added to, this would be the place to make that change. ### refreshUrls The service will also do the more general work of performing all necessary stringTemplating. It takes a single `String tenantId`. When called, either manually via the Controller (see [below](#controller)) or timed task, it will first grab the current time, and the last time this method was called, stored in a "registry" AppSetting. Initially the method looks up all StringTemplates which have been updated since the last time, and if that list has size >0, it runs a full refresh of every PTI, using the method `generateTemplatedUrlsForErmResources` (see [below](#generatetemplatedurlsforermresources)). If that list is empty, it instead first calls `generateTemplatedUrlsForErmResources` for all changed PTIs, then all changed Platforms. This may result in some work being done twice, but this process is fast enough that it's probably going to be fine. These fetches are batched to avoid long lists in memory. As mentioned above, this method is called on a timer, currently set to 2 minutes, so data shouldn't in theory be more than 2 minutes out of date. ### generateTemplatedUrlsForErmResources This method is the heart of all of the refresh work. It takes in a `String tenantId` and a `Map params`, of shape: ``` [ context: 'pti', id: 'abcde', platformId: '12345' ] ``` The available contexts being `pti`, `platform` and `stringTemplate`, with `id` only being necessary for `pti` and `platform`, and `platformId` only being necessary for `pti`. When this method is called, it sets a mutex boolean to `true`. This signifies that it is running, and any more calls until it is finished are added to a queue instead. A tiny amount of "smart" logic dictates that if a `stringTemplate` refresh is added to the queue, we can ditch the rest of the queue. When called with params `[context: 'stringTemplate']`, the method will batch fetch platforms, and for each of those batch fetch PTIs on that platform, then for each of those call `generateTemplatedUrlsForPti` (see [below](#generatetemplatedurlsforpti)). If instead it is called with params `[context: 'platform', id: '12345']`, it will just batch fetch the PTIs for that platform and for each of those call `generateTemplatedUrlsForPti`, and if called with params `[context: 'pti', id: 'abcde', platformId: '12345']` it will call `generateTemplatedUrlsForPti` for that PTI alone. Finally, it will set running to false, and check if anything exists in the queue. If there is, then it will call itself with the first item in the queue. ### generateTemplatedUrlsForPti This method does the actual templating for a given PTI. It takes a `List pti`, which is expected to be of the form `[, ]`, a Map of the relevant StringTemplates for that PTI, of the same shape as given by `findStringTemplatesForId`, and finally the `platformLocalCode`. This method first fetches the PTI with the correct id, then grabs a list of the templatedUrls on that PTI. It then builds the binding required by `performStringTemplates`, and performs the StringTemplating, saving that to another list. These two lists are compared, to see if anything has changed. If it has, then all the existing templatedURLs are deleted from the PTI, and new ones are saved in their places. If not, nothing happens. This is to reduce database churn. ## Controller The final piece of the jigsaw puzzle is the StringTemplateController. Besides allowing for the usual GET/POST/PUT/DELETE operations, it also contains a method to refresh the templatedUrls. The endpoint `erm/sts` acts as the GET/POST for StringTemplates, while `erm/sts/{id}` is for PUT/DELETE of StringTemplates, as per most domain classes we expose. However, the endpoint `erm/sts/template` will call the `refreshUrls` method (see [above](#refreshurls)), triggering a refresh relevant to anything updated since the last refresh. This endpoint is also triggered periodically by a timer task in the ModuleDescriptor: ``` { "permissionsRequired" : ["erm.sts.template"], "methods": [ "GET" ], "pathPattern": "/erm/sts/template", "unit": "minute", "delay": "2" } ``` A GET to the endpoint `erm/sts/template/{id}` will instead return the list of StringTemplates relevant for that id, grouped by context. As mentioned above, there are new permissions required for this controller, which are: `erm.sts.view` -> allows viewing of StringTemplates `erm.sts.edit` -> allows everything from `erm.sts.view`, creating and updating StringTemplates `erm.sts.manage` -> allows everything from `erm.sts.edit` and deleting StringTemplates `erm.sts.template` -> Allows the user to perform string templating ## Speed As things stand, with 15 platforms and around 5000 PTIs, the full process seems to take about 20 seconds. With a higher percentage of those actually needing updates, or more PTIs/Platforms, this time will likely increase. ## Future expansibility This approach leaves open the possibility to manipulate strings from other areas in future, or to add more complicated contexts/rules. One thing that may need to be added to accommodate that is that instead of passing `id=f311d130-8024-47c4-8a86-58f817dbefde` to `findStringTemplatesForId` we may need to pass `id=Platform:f311d130-8024-47c4-8a86-58f817dbefde`, and then switch on that initial class, if we want `Platform` and `TitleInstance` to do the url-based-templating, but say `SubscriptionAgreement` and `Organisation` to ignore the url contexts and focus on some others. One key plan here was to remain as decoupled as possible, in order to allow us to expand/decrease/remove this feature as painlessly as possible in future. It may become necessary to add endpoints which can specifically force through an update on a given PTI/Platform, and it may even be necessary to use context/scopes to be more smart about which Platforms get updated when a StringTemplate update happens, although it's worth noting that that would add to the issues with repeating work mentioned above. There are a fair amount of methods here that seem to do very similar things, or break down more granularly than necessary, but this leaves us very open to adding in optimisations and/or expansions in future.