S3 Import plugin

An Amazon Web Services (AWS) S3 bucket operates as an additional ingestion point into Transact. With the S3 Import plugin, your batch class can be configured to import files from an S3 bucket. Once the plugin is configured, the S3 bucket is monitored every minute for new files. When new files are detected, it uploads the files into your specific batch class. The file is removed from the S3 bucket after processing. The batch class and S3 bucket connection create an efficient processing workflow of large file volumes in Ephesoft Transact.

The plugin applies to batch class level configurations in Transact and should be configured for a one-to-one batch class to S3 configuration. If configuring multiple batch classes for use with the S3 Import plugin, your S3 bucket or S3 bucket and prefix must be unique for each batch class.

Prerequisites

Download the S3 Import plugin from the Customer Support portal.

AWS account requirements

To use the S3 Import plugin, you need to provide your AWS Key ID and AWS Key Secret in Configure the S3 Import plugin. We recommend setting the AWS Identity and Access Management (IAM) policy with restricted S3 access to list buckets, read a specific file, and delete an S3 file.

Refer to the following sample IAM policy:

{ 
   "Version": "2012-10-17",
   "Statement": [ 
      {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": [
            "s3:GetObject",
            "s3:ListBucket",
            "s3:DeleteObject",
            "s3:HeadBucket"
         ],
         "Resource": "*"
       }
    ] 
}

For more information about S3 buckets, see Working with Amazon S3 Buckets. For more information on working with IAM policies, see Identities (Users, Groups and Roles).

Import the S3 Import plugin

This topic only applies to Transact on-premises. For cloud environments, contact Support for help with importing the plugin.

Before configuring the S3 Import plugin, you need to complete the following custom plugin process:

In Transact, go to the System Configuration screen.
Select Workflow Management.
Drag and drop the s3-import-plugin.zip file into the Import Plugin section.
Restart Transact.

The custom plugin is now available for use.

Add the S3 Import plugin

To add the S3 Import plugin to your Transact instance:

Go to the Batch Class Management screen.
Double-click and open the batch class.
Go to Modules > Folder Import.
In the Associated Plugins column, select S3_IMPORT_PLUGIN.
Click the right arrow icon to add the plugin to the Selected Plugins column.

The order of plugins does not matter.
Click Deploy.

Configure the S3 Import plugin

After adding this plugin, configure the S3 Import plugin in your batch class.

Go to Modules > Folder Import.
Select the S3_IMPORT_PLUGIN folder.
Enter the name of the S3 bucket in S3 Bucket Name.

This is a globally unique identifier.
For AWS Key ID and AWS Key Secret, enter your AWS credentials.
Select your AWS Region.
(Optional) If using an S3 bucket prefix, enter the prefix in the Prefix field.

The S3 bucket name and prefix must be unique per batch class.
For Batch Document Count, enter the number of documents to be processed in a batch.

The recommended amount is 20 documents per batch.

If the average size of the files is less than 440 KB, then it is recommended to increase the batch document count for better performance.
For Total Batch Size, enter the file size for the batch class.

The recommended size is 100 MB.
Click Apply to update the batch class.
Click Deploy to deploy the workflow.

You have successfully configured and deployed the S3 Import plugin for your batch class. Your files are now automatically uploaded into your batch class from the AWS S3 bucket.

S3 Import plugin limitations

The following are functional limitations for the S3 Import plugin:

Supports bucket encryption SSE-S3 and unencrypted buckets. SSE-KMS and other types of encryption are not supported, and therefore are not ingested into Transact.
Supports ingestion of S3 objects under the root bucket or one level prefix objects. Nested prefix objects are not ingested into Transact.
When importing or exporting a batch class, the S3 Import plugin configuration is included. You need to re-visit the S3 Import plugin configuration to ensure correct setup of the plugin for the batch class.

See Supported file types for import for the list of supported file types.

Troubleshooting

For cloud support, refer to the following table. Contact Support for further troubleshooting assistance.

Issue	Possible root cause and workaround
There are missing files from Transact.	This may be due to an overlapping ingestion due to more than one batch class ingesting from the same S3 bucket. Verify and ensure that your S3 bucket and optional prefix are unique to the batch class. Otherwise, ingestion may be occurring in another batch and resulting in files missing from your batch instance.
The files are left in the S3 bucket. No ingesting is occurring. No new batch instances are displayed in Transact.	Ensure the following requirements are met: S3 Bucket name exists S3 Prefix - if being used Credentials are correct Read and delete IAM policy rights for account
Batch instances are created, but there are some files that remain in the S3 bucket.	Allow time for files to process as they may be split into multiple batch instances. If the same files are included in each batch, check the IAM policy rights to ensure that there are permissions to delete items from the S3 bucket.

Issue

Possible root cause and workaround

There are missing files from Transact.

This may be due to an overlapping ingestion due to more than one batch class ingesting from the same S3 bucket. Verify and ensure that your S3 bucket and optional prefix are unique to the batch class. Otherwise, ingestion may be occurring in another batch and resulting in files missing from your batch instance.

The files are left in the S3 bucket. No ingesting is occurring. No new batch instances are displayed in Transact.

Ensure the following requirements are met:

S3 Bucket name exists
S3 Prefix - if being used
Credentials are correct
Read and delete IAM policy rights for account

Batch instances are created, but there are some files that remain in the S3 bucket.

Allow time for files to process as they may be split into multiple batch instances.

If the same files are included in each batch, check the IAM policy rights to ensure that there are permissions to delete items from the S3 bucket.

For on-premises support, refer to the following list of error messages that may be logged in the dcma-all.log in Transact for the S3 bucket and prefix-related issues.

Error message	Possible root cause and workaround
<Bucketname>:S3 Bucket does not exist for S3 Import Plugin associated with Batch Class :<batch class name>.	This error message displays when the entered bucket name for the batch class does not exist. Configure a bucket name that exists for the batch class.
Wrong bucket name <batch class name> OR prefix <prefix name> for S3 Bucket <bucket name> associated with Batch Class : <batchClassName>	This error message displays when the S3 bucket name or prefix is not valid. Ensure that the S3 bucket name or prefix is valid and matches your batch class.
AWS Access Key ID is invalid for S3 Import Plugin associated with Batch Class:<batch class name>.	This is related to an S3 access issue. The AWS Access Key ID for the batch class is invalid for the S3 Import plugin associated with the batch class. Enter a valid AWS Access Key ID.
AWS Access Key Secret is invalid for S3 Import Plugin associated with Batch Class: <batch class name>.	The AWS Key Secret is invalid with the associated batch class. Enter a valid AWS Key Secret.
Duplicate batch classes:<batch class name> with the same bucket and prefix are configured.	When more than one batch class has the same bucket name and prefix, then this error displays. Enter a valid S3 bucket name and prefix that is unique to your Ephesoft Transact instance in the other batch classes.