Tree Modes

Tree Modes is a setting that defines how the tree for applications is populated. The mode can be changed at the application and component levels. If you change the tree mode for a component, all its child elements are populated using the selected mode.

The Tree Modes option is available either at the top of the editor view along with the Input, Output, and Variables settings, by right-clicking an application tab title, or a particular component in the Recorder View. If you change the tree mode for a particular component, the asterisk in the finder's Component field is substituted with the selected tree mode. The der_tree_mode property of the component tag is filled with one of the following values: None, ISA, Windows, Auto. To change the component tree mode back to the application-level mode, insert an asterisk in the Component field.

Kofax RPA provides several ways to populate the application tree. By default Kofax RPA detects the type of application the robot is working with (such as a Windows application, terminal, built-in browser, and so on) and automatically forms the tree for this application. Also, Tree Modes helps you select specific options for automated applications.

No Tree: The Robot does not populate the application tree. Use this option for applications that become unstable if a robot tries to extract their tree.
ISA: Stands for Intelligent Screen Automation. This option turns on user interface (UI) recognition that determines interface elements from a graphical representation of the program interface. You can use this mode with programs that do not provide automation API.
Windows: Provides some options for the widget tree generated using Windows automation API.

Intelligent Screen Automation (ISA)

ISA provides extended screen recognition capabilities by automatically finding UI elements and shapes.

Do not use Intelligent Screen Automation (ISA) in Attended Automation.

ISA helps you automate applications with limited or no automation API, such as Citrix. Also, you can use this option to automate applications that are otherwise slow, such as Remote Desktop applications, SAP, and others. In general, you can use this option for any application for which the application tree is incorrectly populated or is otherwise difficult to create a finder for.

Each recognized UI element contains several attributes that you can use in your finders.

Common attributes

der_x, der_y, der_width,der_height: Coordinates and size of the element.
lt_16_5, rb_15_4: Calculated numbers relative to the element's left top and right bottom coordinates. Element coordinates might change in different program state. For example, selected option with bold text can have different coordinates than the cleared option, and therefore finders might work incorrectly. This property eliminates coordinate change issues and can be used to create reliable finders. If a UI element has a name property, it is used in a finder by default when you insert an action step. Otherwise, property lt_16_5 is used in a finder.

The following table lists supported UI elements.

UI element name	Widget name	Element-specific attributes	Notes
Dialog box, frame, pane, and other	container	N/A	A general parent element that contains other elements. For example, an entire dialog box or a form inside this dialog box can be a container. One of the special containers that Kofax RPA can recognize is a table.
Element	element		A general UI element that does not contain other elements.
Icon	icon	label	A graphical element that does not contain any text.
Table	table		Tables are represented in the application tree as separate elements with the following restrictions: merged cells are not recognized and headers are the same as regular cells. Each cell is defined as an item in the application tree. Items can contain other UI elements, such as check boxes, text boxes, and other.
Row	row		A row in the table tree.
Item	item		An item within a row in the table tree.
Text box	textbox	label	A text field.
Check box	checkbox	label	A check box.
Option button	radiobutton	label	An option button or radio button.
Text and text label	linklabel	name inverted	This is either a standalone text element, such as dialog box subtitle, or a UI element label, such as a check box label, text box label, and so on. name: If a text is recognized as a UI element label, the UI element will contain a label property with the label name as its value. In the following example, textbox element will contain label="Name" property. inverted: If the detected by OCR text is dark on a white background, inverted is set to 0. If the text is white on a dark background, inverted is set to 1.

ISA tips and tricks

Recognition results depend on many factors, such as screen fonts, background and foreground color, text size, and so on. If you want to improve recognition results, try using different color schemes in your interface, such as use regular monospaced black-colored fonts on a bright background and larger font size.
Selected text with inverted colors in a text field might not be correctly recognized. To improve recognition results, remove the selection from the text field (such as by clicking somewhere else) before extracting its value.
When you work with dynamic content forms and text fields that might be recognized differently in different state, use the Freeze Tree step to hold the application tree state and improve recognition results.
If you want to extract value from an element, use the Extract Value From action as you would do when using Windows automation API. If the element's value is not recognized in ISA mode, try Extract Text From Image action.
You can edit extended OCR settings in the ocr.cfg file. See Extended OCR Settings for more information.
You can train Tesseract to recognize your character set using either TTF fonts or UI screen shots. See Train Tesseract for more information.
Before automating SAP application, set System Dependent Theme in the SAP GUI Options for better recognition results.

If you connect remotely to a Windows automated device via Remote Desktop Protocol (RDP), resolution and color depth depend on the network connection speed between your computer and remote device. Different connection speed in different robot runs can lead to issues if robots use Image Finder and Intelligent Screen Automation (ISA). For Image Finder and ISA it is very important to always receive the same picture (pixel-by-pixel) in the same application state. Always use the same explicitly-specified resolution and color depth parameters (g: desktop geometry (WxH) and a: connection color depth) for the RDP connection in the Open step. For example:

rdp://user1:MyPassword@MyDesktop?g=640x480&a=16

Change or add UI recognition language

By default ISA uses a language specified in the ocr.cfg file. If you want to add or change the UI recognition language, see Change OCR engine and language topic. If you want to change the ISA UI recognition language for a specific computer, perform the following steps.

In the text editor, open the isa_v1.cfg file located in the lib directory as follows.
- On the Windows-based automated computer with installed Desktop Automation Service:
  
  C:\ProgramData\Kofax RPA\<build number>\lib\tessdata.
  
  If you have several versions of the Desktop Automation Service installed on the automated computer, locate the version of the service that automates the desktop using ISA. See the "Windows tab" in the Configure Desktop Automation Service topic for details.
- On the local Windows-based computer to use with built-in browser:
  
  nativelib\hub\windows-x64\<build number>\lib* in the Kofax RPA installation directory. Example:
  
  C:\Program Files\Kofax RPA 11.3.0.0\nativelib\hub\windows-x64\622\lib
- On the local Linux-based computer to use with built-in browser:
  
  nativelib/hub/linux-x64/<build number>/lib in the Kofax RPA installation directory. Example:
  
  Kofax_RPA_11.3.0.0/nativelib/hub/linux-x64/533/lib
* The build number is different in different versions of the program.
In the ocr_language parameter either replace default with a language of your choice or, if you want to use more than one language, add another language using the plus sign, such as ocr_language=eng+jpn. Save and close the file.
Restart the Desktop Automation Service for the changes to take effect.

To return to the default recognition language, perform the steps above specifying default in the ocr_language parameter.

Add UI recognition language for Tesseract

If you use Tesseract OCR engine, ISA can automate the English language UI by default. Use the following procedure to add more languages for UI recognition. See changing the default OCR language for information on changing the recognition language. Note that using more than one language simultaneously for screen recognition slows down robot execution and deteriorates recognition results.

OmniPage OCR engine includes all supported languages in the Kofax RPA installation.

Download the .traineddata file for the required language from the https://github.com/tesseract-ocr/tessdata. For example, the file for the Japanese language is jpn.traineddata.

Make sure downloaded .traindata file is compatible with your version of the tesseract.dll file. For compatibility information, see the README file published at the bottom of the traineddata download page at https://github.com/tesseract-ocr/tessdata.
Copy downloaded trained data file to the appropriate folder:
- On the Windows-based automated computer with installed Desktop Automation Service:
  
  DesktopAutomationService\lib\tessdata in the Desktop Automation Service installation directory. Example:
  
  C:\Program Files\Kofax RPA DesktopAutomation 11.3.0.0\DesktopAutomationService\lib\tessdata
- On the local Windows-based computer to use with built-in browser:
  
  nativelib\hub\windows-x64\<build number>\lib\tessdata* in the Kofax RPA installation directory. Example:
  
  C:\Program Files\Kofax RPA 11.3.0.0\nativelib\hub\windows-x64\622\lib\tessdata
- On the local Linux-based computer to use with built-in browser:
  
  nativelib/hub/linux-x64/<build number>/lib/tessdata in the Kofax RPA installation directory. Example:
  
  Kofax_RPA_11.3.0.0/nativelib/hub/linux-x64/533/lib/tessdata
* The build number is different in different versions of the program.
Change the UI recognition language as described in the "Change or add UI recognition language" section above.

Windows Options

To generate a widget tree for Windows applications, Kofax RPA uses the UI Automation framework built into Windows. Also, Kofax RPA provides some extended support for particular applications. If you experience any issues working with applications using Kofax RPA extensions, turn them off by clearing the selection in the Windows tree mode.

Excel

Turns on extended support for Microsoft Excel, such as to retrieve cells from the spreadsheet. Excel support in Kofax RPA is also extended with Built-in Excel Driver features.

When using Enter Text in the Excel on a remote device, do not leave the Excel spreadsheet in edit mode (the word Edit appears in the lower-left corner of the Excel program window) if you want to work with it in the following steps. To exit the edit mode, perform one of the following.

Press ENTER using the Press Key step. Excel exits Edit mode and selects the cell directly below the current cell.
Press TAB using the Press Key step. This stops Edit mode and selects the cell to the right of the current cell.
Click a different cell.
Press F2 using the Press Key step.

See Microsoft documentation for more information.

Internet Explorer

Turns on extended Internet Explorer support to retrieve DOM (Document Object Model) tree as it provides a more accurate result in the application tree. Note that Kofax RPA also provides a built-in chromium web browser to access websites. Note the following when considering using extended Internet Explorer support.

If extended Internet Explorer support is enabled and the robot detects an Internet Explorer window:
- A piece of Java-script is injected that traverses the Document Object Model (DOM) and creates a JSON representation. This runs in the context of the Internet Explorer page.
- The JSON is placed into an input tag with id "kapowDataElement" in the page itself.
- The robot parses the JSON content and creates the sub-tree from that content.
If extended Internet Explorer support is disabled, the UI Automation is used to get the object tree from Internet Explorer (this is the same tree that you get using Microsoft's Inspect.exe).

Java

In some situations the Java Access Bridge does not work and it can help to switch to legacy mode by clearing the selection of this option. The Java option can also have an influence on Java Applets in Internet Explorer.

SAP

SAP is handled differently from other Windows applications. Working with SAP depends on scripting API that might by slow. You can turn off SAP support by clearing this option.

Hidden

Specifies whether to extract the entire tree of an application. This option is selected by default. If you clear the selection, Kofax RPA skips elements that are reported as off-screen, such as list boxes or tables with many elements. Clearing the selection reduces the time needed to extract the tree.

Legacy Depth+Index

Adds the "depth" and "index" attributes to the application tree that were used in Kofax RPA 10.2. If you have robots created in Kofax RPA version 10.2 that use component finders with "depth" and "index" attributes, select Legacy Depth+Index for the robots to work in Kofax RPA 10.3 and later.

Max Depth: Sets the maximum number of nested elements each node shows for all application windows in the view.
Max Siblings: Sets the maximum number of sibling elements each node shows for all application windows in the view.
Specify 0 for no restrictions.

Excel API Retry Count

Specifies the number of retries when accessing the Excel window via Windows API. You can increase the number if Excel response is slow.

Excel API Retry Wait Time

Specifies the time in milliseconds between retries when accessing the Excel window via Windows API.