OpenSearch with CP4BA Filenet - Indexing and Search enhancements

GDA is the new common document and archive platform for the belastingdienst.

GDA uses OpenSearch instead of  (or in addition to) Filenet DB queries when searching for documents. GDA functionality requires to search by any property - and business users can define any additional property they see as relevant for their documents (we have a BAW solution which modifies the datamodel/creates subclasses in Filenet almost FN admin intervention). So instead of creating DB indexes for all properties, GDA uses the OpenSearch functionality to also query for date ranges, integer values etc.  

Enhancement request for indexing:

1. existing integration to OpenSearch is missing the ability to index also other types of properties (CBR enabled property can only be a String property)
2. ability to define mapping (currently Filenet doesn't define any datatype mapping - since all properties are just strings). And for String properties it should be possible to define nested or flat data type (e.g. if the string property stores JSON value)
3. ability to NOT index the content of the document - just properties  (or ability to add it to different index)
4. ability to index some documents synchronously 

Enhancement request for searching:

1. ability to execute complex openSearch query (especially nested query for nested datatypes) instead of just simple queryString.

Potential solutions and alternatives for GDA:

    1. and 3. can done if it would be possible to define ingest pipeline for openSearch.  it's 1 additional query parameter which can be added using nginx rewrite rule or configure the Filenet to go to openSearch via AppConnectEnterprise.

 4. currently the only way to implement is to create own event action that will call openSearch to index document... and than we can cover also 1.-3. with our custom queue sweep.

    5. Product Development wrote that they are preparing (for next release in june 2025) a new feature that will allow: "send queries to the engine based on the parameters in the expression (of which there can be arbitrarily many) and return results which are then combined with the database query to produce properly security-filtered final results."  Something like CONTAINS function but more generic with following construction: SELECT … FROM DTO::MySpecialSearch(Document, <parameters>) WHERE <property conditions>.  So it will allow us to execute complex openSearch query (not just passing queryString) and combine the results with Filenet DB query and leverage Filenet security.

    5b. alternative is that we will not execute Filenet query but directly openSearch query and either add security to OpenSearch or merge the results with Filenet query to have property security-filtered results.

    5c. we've been also considering to use searchPipeline (search request processor), but the script processor doesn't allow to change the queryString to nested query - but again if we will "intercept" the openSearch query request in ACE we can do whatever we want.

Discussed with Development SME is an alternative to use DependentObjects instead of JSON values encoded in String properties, but that seems a slower solution for ingestion and may also have some performance challenge when searching because it needs to join with 2 additional tables (and one is ListOfObjects where huge number of records may be stored and using multivalue properties may impose a bottleneck)

The evaluation of the alternatives by DTO is not completed yet. DTO already created and is using the model where complex properties are stored as JSON in String properties.

This idea is interesting to not only apply this on complex properties, but also on additional properties defined by business. If each subclass will have these additional properties defined by business in own dependent object (DB table) than it won't affect docVersion index and we can create index for all properties.