Vertical scaling of import sources

TotalAgility supports optimized scaling of import sources ingestion threads for a multi-tenant Import Connector. This helps optimize ingestion performance and throughput for tenants, without scaling out the system. This is especially useful for large tenants with higher scaling requirements and helps in the following ways:

Scalability: Enables vertical scaling of import sources without scaling out the entire TotalAgility Core Worker cloud service.
Performance: Improves fault tolerance and ensures large tenants can vertically scale ingestion import sources.
Consistency: Ensures that smaller tenants maintain their throughput and address the requirements of larger tenants, who require higher throughput.

To allow configuring vertical scaling for import sources, the following options are added:

In the Tenant Management System:
- Maximum Ingest threads: The maximum ingest threads allowed for a tenant. This limit defines the maximum concurrent polls allowed for an import source in the Import connection in TotalAgility Designer.
- Override Maximum Ingest threads: Overrides the maximum ingest threads allowed for a tenant.
In TotalAgility Designer:
Max concurrent polls: The maximum number of concurrent polls allowed.

There are two modes of local import connection:

Storage mode: The storage mode is used if a local import connection is configured in storage mode. Retrieves documents from various sources and saves them in internal storage, which is available for retrieval using a web service interface. TotalAgility connects to the Message Connector web service interface and retrieves the documents for import. This mode is recommended for larger jobs.
Direct mode : : The direct mode is used if a local import connection is configured in direct mode. It is also used with shared connectors.

This topic describes a simple deployment, where each mode uses one Core Worker and one Message Connector.

For additional hints on using multiple Core Workers and/or multiple Message Connectors, see "Horizontal scaling hints".

Relevant configuration values for the storage mode

Value	Description	Location of the value
PrefetchSize	The number of messages prefetched by the Message Connector from passive inputs. ( Default: 1) If this limit is reached, input is halted until messages are imported.	Message Connector Configuration > General > Prefetched messages
IngestTaskInterval	Defines the interval of calling the Ingest System task. We recommend keeping the default values.	For the "Local connectors" connector type: Designer > System > System tasks > Ingest Local Connectors. For the "Shared connectors" connector type: Designer > System > System tasks > Ingest Shared Connectors.
MaximumNumberofMessagesToImport	The maximum number of messages to import for document ingestion. (Default: 50)	Agility.Server.Core.WorkerService.exe.config
IngestSystemTaskThreadPoolSize	The size of the thread pool for the IngestSystemTask. (Default: 4) Example: If the size is set to 5, only a maximum of five threads can be used for import operations. Thread pools are typically shared with other tasks, so these threads may not be available exclusively.	Agility.Server.Core.WorkerService.exe.config

How does the message import work in a Storage mode?

Import of messages in storage mode depends on the following parallel activities:

Activity 1: Transfer messages from import sources to Message Connector:
Message Connector uses one thread for each configured passive input to transfer messages from the input to the Message Connector storage. These threads are paused if the number of due messages in the inbox (excluding those already locked for import by Activity 2) reaches PrefetchSize.

Additional threads are used for active inputs, like reception from SMTP and FoIP Web Service. PrefetchSize does not affect these inputs.

All active and passive inputs are paused if the storage filled by active messages reaches 80%.
Activity 2: Transfer and convert messages from Message Connector into Tungsten Capture.
The Ingest system task is activated every minute by default (interval is defined by IngestTaskInterval). It parallelly, polls all configured (active) Message Connectors and locks new messages until no further messages are available or the MaximumNumberofMessagesToImport limit is reached.

The locked messages are imported by up to IngestSystemTaskThreadPoolSize worker threads. This import operation consists of the following major steps:
- Get the message with optional conversions from the Message Connector.
- Save the message in the TotalAgility system.

The 1^st step is usually faster but takes longer than the 2^nd step if documentation conversion is included.

Configuration hints

Keep the value of PrefetchSize as low as possible. A higher value may improve the performance in some cases, but it can also increase the number of messages lost if the storage file is lost.
The MaximumNumberofMessagesToImport value typically limits the maximum number of messages that can be imported with a single IngestSystemTask call. A higher value can improve the performance, but it may also increase the number of messages lost if the storage file is lost.
Increase the value of IngestSystemTaskThreadPoolSize to improve the performance; however, we do not recommended a value of above 5 if many messages are imported with document conversions.
During document conversion, the maximum number of concurrent document conversions by all Message Connectors on the same machine is typically limited to three. Additional conversion requests are queued in the Message Connector, reducing the overall performance.

Horizontal scaling hints

If multiple on-premise, on-premise multi-tenant, or Azure core workers are used, the "System Task Ingest" locks the next messages randomly on any core worker. This means the number of messages that can be imported within one IngestTaskInterval is still limited to MaximumNumberofMessagesToImport.
Higher values of PrefetchSize, or MaximumNumberofMessagesToImport may reduce the load distribution between instances.

Relevant configuration values for Direct mode

Value	Description	Location of the value
MaximumDirectIngestThreads	Maximum number of direct import threads per tenant and core worker. (Default: 1)	One of the following locations with decreasing preference: The Override Maximum Ingest Threads tenant-specific setting is enabled. See "Configure the general settings for a tenant" in the Tungsten TotalAgility Tenant Management System Help. If a service profile is assigned to the data center, it defines the maximum direct ingest threads. See "Manage Service profiles" in the Tungsten TotalAgility Tenant Management System Help. Agility.Server.Core.WorkerService.exe.config
DirectIngestBurstDurationInSeconds	Maximum duration since the start of a worker thread where it continuously imports the available messages. (Default: 600 seconds (10 minutes)) This value usually terminates the MaximumDirectIngestThreads value if no input is due for polling. If this duration is reached, the worker thread is completed after the current import/poll operation.	Agility.Server.Core.WorkerService.exe.config
IngestSystemTaskThreadPoolSize	The size of the thread pool for IngestSystemTasks. For example, if the size is set to 5, only a maximum of five threads can be used for import operations. (Default: 4) Thread pools are typically shared with other tasks, so these threads may not be available exclusively.	Agility.Server.Core.WorkerService.exe.config
MaxConcurrentPolls	The maximum number of concurrent threads for an input per Core Worker.	Designer > Integration > Import Settings > Connection > Import Source > General > Max concurrent polls
IngestTaskInterval	It defines the interval of calling the Ingest System task. We recommend that you to keep the default values.	If the "Local Connectors" connector type is selected, Designer > System > System tasks > Ingest Local Connectors. If the "Shared connectors" connector type is selected, Designer > System > System tasks > Ingest Shared Connectors.

How does the message import work in a Direct mode?

The Ingest System Task is activated every minute by default (interval is defined by IngestTaskInterval).
The system task queues a new worker thread if all the following conditions are true:
- At least one input is due for polling. If MaxConcurrentPolls is greater than 1 (2, 3, and so on) and if the input is continuously available for at least 50 (100, 150, and so on) seconds, it is considered as due for polling with a 2^nd (3^rd, 4^th, and so on ) thread.
- The number of active or queued worker threads (for the current tenant) is lower than the MaximumDirectIngestThreads configuration value.
- The total number of active or queued worker threads for all tenants is lower than the size of the thread pool for SystemIngestTask (worker tasks) and IngestSystemTaskThreadPoolSize.
Each worker thread imports messages until one of the following conditions is true:
- No further (unlocked) messages are available.
- The DirectIngestBurstDurationInSeconds duration is reached.
Usually, one worker thread is used to poll all inputs; if this worker thread is unable to complete all polls within the system task interval (default: 1min), one additional worker thread is added every minute until one of the following limits is reached:
- Each mailbox with permanent input has MaximumIngestImportThreads worker threads.
- The total number of worker threads for a tenant is MaximumDirectIngestThreads.
- The total number of worker threads for all tenants is IngestSystemTaskThreadPoolSize.

Configuration hints

The actual number of worker threads for a single input is limited to MaxConcurrentPolls.
The actual number of worker threads for all inputs of a tenant is limited to MaximumDirectIngestThreads.
The actual number of worker threads for all input of all tenants is limited to IngestSystemTaskThreadPoolSize.
Optional. Use additional threads according to MaximumDirectIngestThreads and MaxConcurrentPolls to improve performance by scaling vertically, but carefully increase the number of threads, especially on multi-tenant systems or when document conversion is used on input.
We recommend keeping the default value of DirectIngestBurstDurationInSeconds. Make changes only in very exceptional cases. An increased value may slightly improve the performance, but it may reduce the distribution of resources between tenants.

Horizontal scaling hints

If multiple on-premise, on-premise multi-tenant, or Azure core workers are used, the check for starting a new worker thread happens randomly on any core worker. This means the time taken to start the 2^nd, 3^rd, and subsequent worker threads for an input on a specific core worker increases with the number of core workers.