Tree Modes

Kofax RPA provides several ways to populate the application tree. By default, Kofax RPA detects the type of application the robot is working with (such as a Windows application, terminal, built-in browser) and automatically forms the tree for the application.

Also, Tree Modes helps you select options for automated applications. For example, while working in the built-in browser, you can change the tree mode to ISA option for a specific element to create a robust finder.

1. If the Tree Modes area is not expanded, click the down arrow, which changes to an up arrow.

2. If no definitions exist, click the Add Tree Mode button or right-click the context menu to select Add Tree Mode.

3. If definitions exist, use one of the following methods:

   • To add after existing entries, click the green plus sign.

   • To add between existing entries, select the preceding tree mode, then select Add Below to insert the new one.

   • To add at the top of an existing list, place the pointer at the top and select the Add Tree Mode option from the context menu.

4. From the drop-down list, complete the following Finders entries:

   1. From the Device drop-down list, select the device.

   2. Type the Application name.

   3. Type the Component name.

5. From the Tree Mode drop-down list, select one of the following modes:

   • ISA: Turns on user interface (UI) recognition that determines interface elements from a graphical representation of the program interface. Use this mode with programs that do not provide automation API.
   • Windows: Provides options for the widget tree generated using Windows automation API.
   • No Tree: Does not populate the application tree. Use this option for applications that become unstable if a robot tries to extract their trees.

Change tree modes at the application and component levels.

• When you change the tree mode for a component, all of its nested elements are automatically populated using the selected mode.

• When you change the tree mode for a component, the asterisk in the finder's Component field is substituted with the selected tree mode.

  The der_tree_mode property of the component tag is filled with one of the following values: None, ISA, Windows, Auto.

• To change the tree mode of a component back to the application-level mode, insert an asterisk in the Component field.

Use the Tree Modes area in the Robot definition pane to delete definitions. See Edit Robot for navigation and context-menu options.

In the Tree Modes area, right-click the device and click Delete.

The ISA (Intelligent Screen Automation) option provides extended screen recognition capabilities by automatically finding UI elements and shapes.

ISA helps you automate applications with limited or no automation API, such as Citrix. Also, you can use this option to automate applications that are slow, such as Remote Desktop applications, SAP, and others.

In general, use this option for any application for which the application tree is incorrectly populated or is otherwise difficult to create a finder for.

Each recognized UI element contains several attributes to use in your finders.

Common attributes

• der_x, der_y, der_width,der_height: Coordinates and size of the element.

• lt_16_5, rb_15_4: Calculated numbers relative to the element's left top and right bottom coordinates. Element coordinates might change in different program state.

  For example, a selected option with bold text can have different coordinates than the cleared option, and therefore finders might not work correctly. This property eliminates coordinate change issues and can be used to create reliable finders. If a UI element has a name property, it is used in a finder by default when you insert an action step. Otherwise, property lt_16_5 is used in a finder.

The following table lists supported UI elements.

ISA tips and tricks

• Recognition results depend on many factors, such as screen fonts, background and foreground color, text size, and so on. If you want to improve recognition results, try using different color schemes in your interface, such as regular monospaced black-colored fonts on a bright background and larger font sizes.
• Selected text with inverted colors in a text field might not be correctly recognized. To improve recognition results, remove the selection from the text field (such as by clicking somewhere else) before extracting its value.
• When you work with dynamic content forms and text fields that might be recognized differently in different states, use the Freeze Tree step to hold the application tree state and improve recognition results.
• To extract value from an element, use the Extract Value From action as you would when using Windows automation API. If the element's value is not recognized in ISA mode, try Extract Text From Image action.
• Edit extended OCR settings in the ocr.cfg file. See Extended OCR Settings for more information.

• Train Tesseract to recognize your character set using either TTF fonts or UI screen shots. See Train Tesseract for more information.

• Before automating SAP application, set System Dependent Theme in the SAP GUI Options for better recognition results.

• If you connect remotely to a Windows automated device via Remote Desktop Protocol (RDP), resolution and color depth depend on the network connection speed between your computer and the remote device. Different connection speeds in different robot runs can lead to issues if robots use Image Finder and Intelligent Screen Automation (ISA).

  • For Image Finder and ISA, it is very important to always receive the same picture (pixel-by-pixel) in the same application state.

  • Always use the same explicitly-specified resolution and color depth parameters (g: desktop geometry (WxH) and a: connection color depth) for the RDP connection in the Open step.

    Example: rdp://user1:MyPassword@MyDesktop?g=640x480&a=16

Change or add UI recognition language

By default ISA uses a language specified in the ocr.cfg file. If you want to add or change the UI recognition language, see Change OCR engine and language topic.

To change the ISA UI recognition language for a computer, perform the following steps.

1. In the text editor, open the isa_v1.cfg file located in the lib applicable directory as follows.

   • On a Windows-based automated computer to use with installed Desktop Automation Service:

     C:\ProgramData\Kofax RPA\<build number>\lib\tessdata.

     If you have several versions of the Desktop Automation Service installed on the automated computer, locate the version of the service that automates the desktop using ISA. See the Windows tab in the Kofax RPA Desktop Automation Service Guide.

   • On a local Windows-based computer to use with built-in browser:

     nativelib\hub\windows-x64\<build number>\lib in the Kofax RPA installation directory.

     The build number is different in different versions of the program.

     Example: C:\Program Files\Kofax RPA 11.5.0.0\nativelib\hub\windows-x64\622\lib

   • On a local Linux-based computer to use with built-in browser:

     nativelib/hub/linux-x64/<build number>/lib in the Kofax RPA installation directory.

     Example: Kofax_RPA_11.5.0.0/nativelib/hub/linux-x64/533/lib

2. In the ocr_language parameter, either replace default with a language of your choice or, if you want to use more than one language, add another language using the plus sign, such as ocr_language=eng+jpn.

3. Save and close the file.

4. Restart the Desktop Automation Service for the changes to take effect.

To return to the default recognition language, perform the previous steps, specifying default in the ocr_language parameter.

Add UI recognition language for Tesseract

If you use Tesseract OCR engine, ISA can automate the English language UI by default. Use the following procedure to add more languages for UI recognition.

For information on changing the recognition language, see "Change default OCR language" in the Kofax RPADesktop Automation Service Guide. Note that using more than one language simultaneously for screen recognition slows down robot execution and deteriorates recognition results.

OmniPage OCR engine includes all supported languages in the Kofax RPA installation.

1. Download the .traineddata file for the required language from GitHub.

   For example, the file for the Japanese language is jpn.traineddata.

   If the files are not available in the link above, search for version 3.04 from the main repository page.

   Tesseract version 3.05.00 is compatible with Kofax RPA 11.5.0. However, RPA uses Tesseract 3.04.00 training files.

   Make sure the downloaded .traineddata file is compatible with your version of the teserract.dll file. For compatibility information, see the README.md file published at the bottom of the traineddata page.

   Using the wrong training data does not result in error messages, but may cause poor OCR results.

2. Copy the downloaded .traineddata file to the appropriate folder.

   • On a Windows-based automated computer to use with installed Desktop Automation Service:

     DesktopAutomationService\lib\tessdata in the Desktop Automation Service installation directory.

     Example: C:\Program Files\Kofax RPA DesktopAutomation 11.5.0.0\DesktopAutomationService\lib\tessdata

   • On a local Windows-based computer to use with built-in browser:

     nativelib\hub\windows-x64\<build number>\lib\tessdata in the Kofax RPA installation directory.

     The build number is different in different versions of the program.

     Example: C:\Program Files\Kofax RPA 11.5.0.0\nativelib\hub\windows-x64\622\lib\tessdata

   • On a local Linux-based computer to use with built-in browser:

     nativelib/hub/linux-x64/<build number>/lib/tessdata in the Kofax RPA installation directory.

     Example: Kofax_RPA_11.5.0.0/nativelib/hub/linux-x64/533/lib/tessdata

3. Change the UI recognition language as described in the previous "Change or add UI recognition language" task.

To generate a widget tree for Windows applications, Kofax RPA uses the UI Automation framework built into Windows. Kofax RPA provides extended support for some applications.

If you experience issues working with applications using Kofax RPA extensions, turn them off by clearing the selection in the Windows tree mode.

Excel

Turns on extended support for Microsoft Excel, such as to retrieve cells from the spreadsheet.

if you want to work with Excel in the following steps, do not leave the Excel workbook in Edit mode (the word Edit appears in the lower-left corner of the Excel program window).

When using Enter Text in Excel on a remote device, to exit the Edit mode, perform one of the following.

• Press ENTER using the Press Key step.

  Excel exits Edit mode and selects the cell directly below the current cell.

• Press TAB using the Press Key step.

  This stops Edit mode and selects the cell to the right of the current cell.

• Click a different cell.

• Press F2 using the Press Key step.

Refer to the Microsoft documentation for more information.

Internet Explorer

When you use Microsoft Edge in Internet Explorer mode, it turns on extended Internet Explorer support to retrieve the Document Object Model (DOM) tree. This mode provides a more accurate result in the application tree. Also, Kofax RPA provides a built-in chromium web browser to access websites.

Be aware of the following when using extended Internet Explorer support.

• If extended Internet Explorer support is disabled, the UI Automation gets the object tree from Internet Explorer (this is the same tree that you get using Microsoft's Inspect.exe).
• If extended Internet Explorer support is enabled and the robot detects an Internet Explorer window:
  • A piece of Java-script is injected that traverses the DOM and creates a JSON representation. This runs in the context of the Internet Explorer page.
  • The JSON is placed into an input tag with ID "kapowDataElement" in the page.
  • The robot parses the JSON content and creates the sub-tree from that content.

Java

In some situations the Java Access Bridge does not work. It may help to switch to legacy mode by clearing the selection of this option. Also, the Java option can influence Java Applets in Microsoft Edge in Internet Explorer mode.

SAP

SAP is handled differently from other Windows applications. Working with SAP depends on scripting API that might be slow. Turn off SAP support by clearing this option.

Nodes for some elements may be unavailable in the application tree. To make all nodes available in the tree:

1. From the product installation folder, download and save the RegSAPSurrogate.reg file on the computer running the Desktop Automation Service.

   The file resides in: {path}\Kofax RPA Desktop Automation 11.5.0.0\DesktopAutomationService\bin.

2. Run the file and accept any warnings.

3. Restart the Service agent.

Hidden

Specifies to extract the entire tree of an application. This option is selected by default. If you clear the selection, Kofax RPA skips elements that are reported as off-screen, such as list boxes or tables with many elements. Clearing the selection reduces the time needed to extract the tree.

Legacy Depth+Index

Adds the depth and index attributes to the application tree that were used in Kofax RPA 10.2.

If you have robots created in Kofax RPA version 10.2 that use component finders with depth and index attributes, select Legacy Depth+Index for the robots to work in Kofax RPA 10.3 and newer releases.

• Max Depth: Sets the maximum nested elements each node shows for all application windows in the view.
• Max Siblings: Sets the maximum sibling elements each node shows for all application windows in the view.

  Specify 0 for no restrictions.

Excel API Retry Count

Specifies the number of retries when accessing the Excel window via Windows API. Increase the number if Excel response is slow.

Excel API Retry Wait Time

Specifies the time in milliseconds between retries when accessing the Excel window via Windows API.