Add a local fuzzy database
Use the procedure in this topic to add a local fuzzy database
Local fuzzy databases are used in a locator or in a script. Ensure the .CSV or .TXT file for the database is formatted correctly.
It is best practice not to exceed the supported number of records for a local fuzzy database. The default number of supported records is set to five million records. Databases may exceed this limit, but more memory is required. However, even with increased memory allocated, processing speed may be decreased. Alternatively, you can use a Remote Fuzzy Database that is managed through the Tungsten Search and Matching Server.
-
On the
Project tab, in the
Configuration group, select
Project Settings.
- Select the Databases tab.
-
In the
Databases group, select
Add.
The Add New Database window is displayed.
-
On the
Add New Database window, and perform the following steps.
- Enter a Database Name.
- From the Database Type list, select Local Fuzzy Database.
- Select OK.
The local fuzzy database properties window is displayed.
-
In the local fuzzy database properties window, configure the following settings.
Setting
Description
Referenced import file (text or csv file)
Select one of the following reference file locations:
- File system:
-
Browse to the desired location of a local fuzzy database. The import process starts automatically when the window is closed, and a message box is displayed that counts the number of imported database lines. One million lines with three fields take about 1 minute to import. (Default: Selected)
- Web:
-
Enter the URL for your local fuzzy database file.
Select Browse URL to ensure the connection to the specified URL is available.
Provide a User Name and Password if authentication is required.
Column Configuration
This table has the following columns. You can rename an entry by selecting the row and then selecting a cell in that row.
- Column name
-
The name of the database column.
- Search
-
If selected for a field, that field is included in database searches.
Choose this value for each field that might be present on a document.
- Filter
-
It is possible to filter the data before searching to reduce the number of possible records. If this setting is selected for a field it is included in the filtering index. Filtering is possible using scripting only.
This column is visible only when Load database index in memory setting is selected and the Database processing setting is set to Advanced in the Optimization group.
Import Options group
Ignore Case
Converts all search and lookup strings to lowercase, effectively ignoring case. (Default: Selected)
First line contains caption
Select if the first record of the import file contains the column headers. (Default: Selected)
Field delimiter
Enter values into this field to specify what characters separate the import file content into individual fields. (Default: ; (semicolon))
- Tab:
-
Uses a Tab as a delimiter in addition to the characters specified in the Field delimiter setting.
Word separation characters
If fields in the database contain compound words, common characters can be specified so each part of the compound word is searched and evaluated separately. (Default: -, (space, hyphen, comma))
For example, using the default settings, the compound word "Diagon-Alley," is treated as two words, "diagon" and "alley" that are searched and evaluated separately.
The separation characters must correspond to the delimiter characters that are defined for OCR.
- Tab:
-
Uses a Tab as a word separation character in addition to the characters specified in the Word separation characters setting.
- Space:
-
Uses a Space as a word separation character in addition to the characters specified in the Word separation characters setting.
Characters to ignore
Enter a list of characters into this field to filter unwanted characters from the input record. When you want to use a field delimiter that may also be a character in the input, such as a comma (,), then you have to use quotes (") to identify the input strings. However, you probably do not want to retain those quotation marks as part of the final results. (Default: ."'! (period, quotation mark, single quotation mark, and an exclamation point))
If you define the quotes as characters to ignore, they are removed. To define a tab or space as characters to ignore, select the corresponding check box.
- Space:
-
Ignores a Space character in addition to the characters specified in the Characters to ignore setting.
Filtering is case sensitive
Ensures the filtering text entered in the script matches an entry in the database. If the cases do not match exactly, no records are returned. (Default: Selected)
Available when at least one field has the Filter setting selected in the Column Configuration group and when Load database index in memory setting is selected and the Database processing setting is set to Advanced in the Optimization group.
Optimization group
Automatic update from import file
Updates the Local Fuzzy Database automatically when the source file is updated. (Default: Cleared)
For more information, see Local fuzzy database automatic updates.
Load database index in memory
Loads the database into memory. (Default: Selected)
For more information, see Load a local fuzzy database in memory.
If you clear this setting, the Filter setting is disabled in the Column Configuration group.
Database processing
Select one of the following settings to determine the level of processing required for your database searches:
- Basic:
-
Choose if the computer where Tungsten TotalAgility is installed does not have a lot of memory or processing power. Selecting this value means the accuracy of your results can be lower than expected, but the time it takes to generate these results may be significantly faster than if you selected the Advanced value for this option.
If you choose the Basic value for this setting, the Filter setting is disabled in the Column Configuration group.
This setting replicates the search behavior from Tungsten TotalAgility 5.0 that enables users to focus on speed instead of accuracy.
- Advanced:
-
Choose this value if you want the most accurate search results. Selecting this value means the accuracy of your results are better than the Basic value for this setting. However, the time it takes to generate these results depends on the size and complexity of your database, the available memory, and the number of processing cores available on your server. If you do not have a lot of memory and multiple processing cores, or your database is very large and complex, the Basic value may be more suitable. (Default: Selected)
If you choose this value, you can further optimize a database locator that uses this database for speed or accuracy.
If you are unsure which value is best for your project, the best practice is to finish configuring the database locator that uses this fuzzy database and then run several extraction benchmarks to compare the results between the two Database processing values.
For more information, see Fuzzy database processing and optimization.
String Substitution
The string substitution table substitutes Search Text with Replacement Text in the document and in the database. It is used to normalize the results of the text search.
If you use the same dictionary in more than one project, you can create a list of string substitutions and export them to use in the other project.
Use the
Add,
Delete,
Modify,
Import, and
Export buttons to manage the string substitutions for this database.
Table Records Preview
Provides a preview of the selected database.
-
Select
OK to save the Local Fuzzy Database.
The properties window is closed and the new Local Fuzzy Database is displayed in the list of databases in the Databases group.
- Optional. Select OK to close the Project Settings window.
- Save the changes to your project.