Configure and perform clustering
Use the procedure in this topic to organize a group of unknown documents.
Clustering takes a set of unclustered documents and groups them together based on their layout, their content, or both. Depending on how your project is configured or what you want to do with the documents once clustered, several configuration settings are available.
Clustering is supported only for test sets, benchmark sets and the training sets.
- Open the Documents window if it is not already open.
-
Open or select the document set and subset that contains the unsorted documents.
The documents in the selected document subset are displayed in the selected view.
-
Right-click on the document subset and select
Clustering.
The Clustering window is displayed.
-
On the
Configure tab, configure the following settings.
Setting
Description
Use project class names
Uses the project class names as the labels for the clusters. (Default: Selected)
Cluster documents with no assigned class only
Performs clustering for those documents that do not have an assigned class. Any documents with an assigned class are ignored. (Default: Selected)
Selecting this setting ensures that you do not slow down the clustering process by including documents that are already in the Classification Set. Any discrepancies in the number of documents loaded for clustering versus the number of document selected could be because of this setting. Clearing this setting overwrites any existing classification results.
Are most of the documents unstructured?
Select one of the following clustering methods:
- Yes (cluster based on content):
-
Documents are clustered based on content. Documents require recognition results for content classification. If no recognition results are available, an error message is displayed recommending the use of the Recognition on-demand setting.
- No/Unknown (cluster based on layout):
-
Documents are clustered based on layout. Documents used for layout clustering must include an image component. If no image is available, an error message is displayed. (Default: Selected)
Recognition on-demand
Enabled only when cluster based layout is used. (Default: Cleared)
When selected, performs content clustering and you are not sure if all of your documents are already processed by the recognition engine. Before the clustering process starts, recognition is performed when the full text results of the images are missing.
If any documents do not have recognition content, they cannot be clustered. As a result, the best practice is to select this setting unless you have performed recognition in a previous step.
Minimum cluster size
You can specify the minimum number of documents required for a cluster to accept the result as a valid cluster. (Default: 10)
Minimum confidence
Select a value to determine how confident the classification of a document has to be before it is assigned to a cluster.
-
Select
Start Clustering.
A progress bar is displayed showing the clustering process. When clustering is finished, the Identify tab is displayed so you continue processing the documents.
-
Identify the documents by assigning them to a cluster and ensuring the cluster has a label.
Three documents are displayed and depending on the configuration, you can assign the displayed document to a cluster by entering a new cluster label or by selecting an existing label. The documents displayed on this tab differ each time you select Continue Clustering, and after several iterations, the clustering process asks for confirmation of suggested clusters.
The following Statistics are available for the first step of clustered documents.
- Total documents:
-
Total number of processed documents.
- Confirmed documents:
-
The total number of documents that you have manually confirmed during the clustering iterations.
For layout clustering, this value includes documents that are confirmed automatically.
- Unconfirmed documents:
-
The number of documents that are identified as belonging to a cluster automatically. These identified documents are based on other documents assigned to a labeled cluster during the identification iterations. These documents require manual confirmation in order to be assigned to a cluster.
- Remaining documents:
-
Number of documents that are not yet assigned to a labeled cluster or are unclustered.
- Labeled clusters:
-
Number of cluster categories that are already labeled.
- Unlabeled clusters:
- The number of clusters that are suggested but unlabeled.
-
For each displayed document, enter or select a cluster label from the
Assign a cluster list.
The document is confirmed and assigned to the labeled cluster. If any other documents are assigned to this same cluster using layout clustering, these are confirmed automatically. Documents using content clustering require manual confirmation in a later step.
- For each displayed document, select Confirm if the suggested cluster is correct. If the suggested cluster is incorrect, select Assign to change the cluster.
- Regularly view the Statistics to view your progress.
-
Optional. Select a
Filter to view documents in one of the following categories:
- No filter:
-
If selected, no filter is applied to the documents displayed. This means that unknown documents, unconfirmed documents, and those documents assigned to unlabeled clusters are displayed. (Default: Selected)
- Unconfirmed documents for labeled clusters:
-
If selected, all documents are displayed in the thumbnail view that were recognized and assigned to a cluster.
- Documents in unlabeled clusters:
-
If selected, only documents that were recognized and assigned to a cluster, but the cluster itself is unlabeled, are displayed.
- Unclustered documents:
-
If selected, only those documents that were not recognized during clustering and could not be assigned to a cluster are displayed.
-
Select
Continue Clustering processing the documents for each step.
More documents are displayed until all documents are clustered. When clustering is finished, you are asked if you want to continue and the Review tab is displayed.
-
Review the clustered documents.
All of the documents in the selected document subset are displayed along with their cluster and status. This enables you to get an overall picture of your clusters. You also have another opportunity to make any changes before assigning a document to a cluster.
-
For any documents in an unlabeled cluster, select a
Cluster Label from the list.
When a label is applied to a cluster, all documents in that cluster are assigned that same label. If the documents are clustered using layout clustering, this is done automatically. For documents clustered using content clustering, manual confirmation is required.
-
If you have added a label for any unlabeled clusters, select
Update Clustering.
Clustering is performed and the documents that belong to the newly labeled cluster are updated accordingly.
- For any unclustered documents, select a Cluster Label from the list.
- For any unconfirmed documents, select Confirm if they are in the correct cluster. If they are in the wrong cluster, select Reassign to assign them to the correct cluster.
-
Once all documents are confirmed, assigned to a cluster, and all clusters have a label, select
Update Clustering.
The Assign tab is loaded.
-
For any documents in an unlabeled cluster, select a
Cluster Label from the list.
-
On the
Assign tab, configure the following settings to decide which of the clustered documents are
assigned a cluster.
Setting
Description
Create project classes from cluster labels
Creates a class for each labeled cluster in the Project Tree. (Default: Cleared)
This is useful if you do not already have the project hierarchy configured and is an easy way to quickly configure a project hierarchy. .
This setting is disabled if your project is read-only.
Minimum content confidence
Enter a value or move the slider to show the minimum content confidence required for a document to be exported. (Default: 0)
Documents with a content clustering confidence below this value are not included in the assign step.
Minimum layout confidence
Enter a value or move the slider to show the minimum layout confidence required for a document to be assigned a cluster. (Default: 0)
Documents with a layout clustering confidence below this value are not included in the assign step.
Confirmed documents
Assigns any documents that were manually confirmed during the clustering process. (Default: Selected)
Unconfirmed documents
Assigns any documents that belong to a labeled cluster but were not manually confirmed during the clustering process. (Default: Selected)
If you select this setting, the unconfirmed cluster is assigned as the classification result, assuming it meets the minimum confidence criteria.
-
Select
OK.
The Clustering window closes and the selected document subset is updated to reflect the cluster labels as their classification result. If Create project classes from cluster labels is selected, the Project Tree is updated with the new classes.