Classification benchmarks
To provide a complete and detailed analysis of classification quality, the Transformation Designer enables you to test the classification against a benchmark document set or the classification training document set. A benchmark provides more information than a preliminary classification test.
A benchmark document set is a set of documents that differs from the training documents and a set of test documents. Each document in a benchmark set has an assigned class. When you generate a benchmark, the classifier compares the assigned class to the classification result based on project-level and class-level classification settings.
You can generate a classification benchmark for all classification settings in a project. This includes all classification settings on the Project Settings - Classification tab, class details, all classifiers, any classification instructions, and any classification script events.
You can also generate a classification benchmark for a specific classifier configured in your project. This means that if you have a layout classifier, a top-level content classifier (Adaptive Feature Classifier), and subtree classifiers, you can generate a benchmark for each. This enables you to compare classifier results and modify classification settings as needed.
You cannot run a benchmark against a normal custom test set because there is no "Assigned Class" value in a test set, and this value is needed for comparison against the classification result. If necessary, convert a custom test set into a benchmark set to generate a benchmark.
When you want to generate a classification or separation benchmark, first store the assigned class structure on disk using the Sort Documents on Disk by Class setting from the Benchmark Sets shortcut menu.
Running a benchmark generates the Result Matrix and displays a grid of the correctly and incorrectly classified documents by class. The Result Matrix provides statistics about unclassified documents and about recall and precision.
For the best results for a project that includes child classes or a complex class hierarchy, select "Classification Benchmark (All Classification Settings)" when generating a classification benchmark for a benchmark set.
Any results that are overridden by exceptions appear in a darker color than the rest of the results. For example, if an added exception is sets a group of unclassified documents to valid, the color becomes dark green.
Depending on whether you generate a classification benchmark for all project classification settings or an individual classifier, the available settings may differ.
Regardless of the document set selected, the classification benchmark results are displayed in a Result Matrix grid. Each class in your project is displayed, and the assigned class is compared to the classification benchmark result. Green results are successful classifications, red results means there are problems with a class, and yellow results indicate unclassified documents. The alignment in the grid indicates the assigned class, so the successful results always are displayed in the middle. Any other results are displayed off-center.
The following toolbar settings are available:
|
Setting |
Name and Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If you toggle off Do not use Project Classification in the toolbar, the following settings are available so you can test an individual classifier.
- Classifier:
-
Select a classifier in the list so you can benchmark its specific settings.
- Minimum confidence:
-
Select the minimum confidence required to assign a classification result.
- Minimum distance:
-
Select the minimum distance between the best and the second best classification result.
You can also perform the following actions using the shortcut menu of a column in the Result Matrix:
- Display in Selection:
-
Highlights the documents in the selected content in the Selection document sent in the Documents window.
- Display in Document Viewer:
-
Displays the first document in the selected column in the Document Viewer. You can use the navigation buttons in the toolbar to navigate between all of the documents in this column.
- Add Benchmark Exception (Valid):
-
Forces the documents in the selected column to be valid. The column color changes to green.
- Add Benchmark Exception (Unclassified):
-
Changes the classification result for the selected column to unclassified. The column color changes to yellow.
- Remove Benchmark Exception:
-
If there are any classification exceptions configured for the selected column, clear these to view the classification result with no exceptions.
In order to optimize the classification quality with the Result Matrix, you can do the following:
-
Generate a classification benchmark for all project classification settings
-
Generate a classification benchmark for a specific classifier
-
View classification benchmark statistics
-
Save classification benchmark results
-
Compare classification benchmark results
-
Add classification benchmark exceptions