Classification benchmarks

To provide a complete and detailed analysis of classification quality, the Transformation Designer enables you to test the classification against a benchmark document set or the classification training document set. A benchmark provides more information than a preliminary classification test.

A benchmark document set is a set of documents that differs from the training documents and a set of test documents. Each document in a benchmark set has an assigned class. When you generate a benchmark, the classifier compares the assigned class to the classification result based on project-level and class-level classification settings.

You can generate a classification benchmark for all classification settings in a project. This includes all classification settings on the Project Settings - Classification tab, class details, all classifiers, any classification instructions, and any classification script events.

You can also generate a classification benchmark for a specific classifier configured in your project. This means that if you have a layout classifier, a top-level content classifier (Adaptive Feature Classifier), and subtree classifiers, you can generate a benchmark for each. This enables you to compare classifier results and modify classification settings as needed.

You cannot run a benchmark against a normal custom test set because there is no "Assigned Class" value in a test set, and this value is needed for comparison against the classification result. If necessary, convert a custom test set into a benchmark set to generate a benchmark.

When you want to generate a classification or separation benchmark, first store the assigned class structure on disk using the Sort Documents on Disk by Class setting from the Benchmark Sets shortcut menu.

Running a benchmark generates the Result Matrix and displays a grid of the correctly and incorrectly classified documents by class. The Result Matrix provides statistics about unclassified documents and about recall and precision.

For the best results for a project that includes child classes or a complex class hierarchy, select "Classification Benchmark (All Classification Settings)" when generating a classification benchmark for a benchmark set.

Any results that are overridden by exceptions appear in a darker color than the rest of the results. For example, if an added exception is sets a group of unclassified documents to valid, the color becomes dark green.

Depending on whether you generate a classification benchmark for all project classification settings or an individual classifier, the available settings may differ.

Regardless of the document set selected, the classification benchmark results are displayed in a Result Matrix grid. Each class in your project is displayed, and the assigned class is compared to the classification benchmark result. Green results are successful classifications, red results means there are problems with a class, and yellow results indicate unclassified documents. The alignment in the grid indicates the assigned class, so the successful results always are displayed in the middle. Any other results are displayed off-center.

The following toolbar settings are available:

Setting

Name and Description

Run Classification Benchmark icon

Perform Benchmark:

Runs a classification benchmark using the selected settings.

Classification Benchmark Type icon

Do not use Project Classification:

Toggles the entire project classification configuration on and off. When off, you can select a specific classifier instead.

Show Unclassified Documents in the Classification Benchmark icon

Unclassified:

Adds an unclassified column to the Result Matrix so you can see any unclassified documents in your selected document set.

Save Project icon

Save:

Displays the Save Benchmark window so you can save a benchmark and compare it at a later time.

Compare Classification Benchmark icon

Compare:

Displays the Classification Benchmark Comparison window so you can compare two saved benchmark files.

Classification Benchmark Info icon

Info:

Toggles additional information about your benchmark results. When enabled, basic statistical information is displayed to the right of the Result Matrix.

Zoom In icon

Zoom In:

Increases the magnification of the current page in the Document Viewer.

Classification Benchmark Statistics icon

Statistics:

Displays a window that contains more detailed statistical data about the precision and recall of the classification benchmark.

Classification Benchmark Exceptions icon

Exceptions:

Displays the Benchmark Exceptions window where you can configure exceptions to your classification results.

If you toggle off Do not use Project Classification in the toolbar, the following settings are available so you can test an individual classifier.

Classifier:

Select a classifier in the list so you can benchmark its specific settings.

Minimum confidence:

Select the minimum confidence required to assign a classification result.

Minimum distance:

Select the minimum distance between the best and the second best classification result.

You can also perform the following actions using the shortcut menu of a column in the Result Matrix:

Display in Selection:

Highlights the documents in the selected content in the Selection document sent in the Documents window.

Display in Document Viewer:

Displays the first document in the selected column in the Document Viewer. You can use the navigation buttons in the toolbar to navigate between all of the documents in this column.

Add Benchmark Exception (Valid):

Forces the documents in the selected column to be valid. The column color changes to green.

Add Benchmark Exception (Unclassified):

Changes the classification result for the selected column to unclassified. The column color changes to yellow.

Remove Benchmark Exception:

If there are any classification exceptions configured for the selected column, clear these to view the classification result with no exceptions.

In order to optimize the classification quality with the Result Matrix, you can do the following:

More info