Add Classification Benchmark Exceptions
Once a project has two or more classes, the Benchmark Exceptions icon becomes available on the Classification Benchmark toolbar.
If you use content classification, you must have recognition results before starting the benchmark. When a classification benchmark using all settings is calculated, the documents in the training set or the selected Benchmark Set are classified using your current settings. When a benchmark for a specific classifier is run, all other classification settings are ignored so you see only the classifier results.
The training set or benchmark set results are compared to the Assigned Class in the document set, and marks mismatches as invalid. Sometimes, the mismatch between the classification and the Assigned Class of a document results in documents being marked invalid, even when they are correctly classified. You can add benchmark exceptions that set invalid results to valid or unclassified. The Classification Benchmark window Result Matrix will then provide more accurate statistics.
For example, several documents have an Assigned Class for a parent class that contains inherited child classes. When the benchmark is performed, the classification result returned is for one of the child classes rather than the parent class. These documents are marked invalid in the Classification Benchmark window Result Matrix, even though the classification result is valid. As a result, even if they are correctly classified as the inherited child class, they are displayed as invalid in the classification benchmark results. You can add an exception so that the invalid child result is valid, but only in the Classification Benchmark window and the Result Matrix.
Also use exceptions when inheritance and nested classes are not the issue. For example, an organization has two classes with similar settings at the same level. Due to service level agreements, you do not want to count misclassified documents between the two classes as false positives. For technical and business reasons, these two classes are commonly mixed up. One class contains draft documents; the other contains final certified versions of the same documents. The only difference between the two documents is that the final certified documents have a signature and certification stamp. These items are difficult to detect, so as a result, all documents classified as either of these two classes are uncertain. To use the Result Matrix to gather statistics, add exceptions to thoroughly test benchmark results.
As each exception is added, the Result Matrix graph, the Document Evaluation results, and the Project Statistic results are automatically updated. Similarly, if a class is deleted from the project, any exceptions using that class are also removed.
You can add benchmark exceptions by following these steps:
- Open the Classification Benchmark window if it is not already open.
-
Generate a benchmark for
all classification settings or a
specific classifier for your classification
training set or a benchmark set.
The selected document set is classified and the benchmark results are displayed in the Result Matrix.
-
In the
Classification Benchmark window toolbar, click
Benchmark Exceptions
.
The Classification Benchmark Exceptions window is displayed.
- In the Benchmark Exceptions window, click Add Exception .
- From the Desired Result list, select a class.
- From the Classification Result list, select a class.
- From the Exception Status list, select a value.
-
Optionally, add a
Comment.
Although optional, the value you enter in this step is displayed as a tool-tip on the Result Matrix for any affected cells.
-
Click
OK.
The Classification Benchmark Exceptions window closes and the benchmark results in the Result Matrix are updated automatically.