Skip to main content

Advanced SDK features for Selenium

Below are possible configuration variables to pass to the SDK at initialization time or runtime to improve tests reliability and performance.

Custom thresholds with custom_ai_threshold​

This extra parameter can be passed to any of the find_element methods (e.g. find_by_ai, find_by_id). When the driver finds the wrong element (e.g. a text element with similar text or size, a similar icon), you can increase the threshold between 0 and 1 to improve the accuracy of the element selection. The threshold passed in is the minimum score that the element needs to get to be returned. If the scores are consistently low, consider adding more training examples by labeling them in the web UI.

1.0 is the maximum and means an exact match with the training data. For hard elements we recommend 0.99.

tip

The threshold does not override exact_match_first_threshold when using do_exact_match_first. These parameters are explained at the bottom of the page.

element = driver.find_by_ai("searchbox", custom_ai_threshold=0.99)
element = driver.find_by_id("searchbox", custom_ai_threshold=0.99)
element = driver.find_by_class_name("searchbox", custom_ai_threshold=0.99)
etc

Javascript Chopper - use_fast_js_chopper​

This argument passed in at initialization time tells the SDK to use Javascript for matching the visual element to a Selenium element. It is faster, but can also return the wrong element in certain cases where the elements have things above them in the z-index.

smart_driver = SmartDriver(driver, "??API_KEY??", initialization_dict={"use_fast_js_chopper": True})

Dynamic Waits, WebDriverWait, By locator​

You can dynamically wait for an element to appear on the page using standard ExpectedConditons and WebdriverWait. For this you can use our By locator superclass. It works like a normal By locator but adds By.ai

Here are some examples


from devtools_ai.selenium import SmartDriver, By, NoElementFoundException
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = SmartDriver(chrome_driver, "??API_KEY??")
wait_time = 10
wait = WebDriverWait(driver, wait_time)
wait.until(EC.presence_of_element_located((By.AI, "flutter_push_button")))
# You can also specify the threshold
wait_time = 30
wait = WebDriverWait(driver, wait_time)
wait.until(EC.presence_of_element_located(By.ai("el_why_change", custom_ai_threshold=0.99)))

Force template matching first to save time do_exact_match_first​

Our system by default uses a ML based model to identify elements on screen, and can sometimes use template matching as a fallback mechanism if the ML is unable to find high scoring elements. For some users, speed is more important than the flexibility of AI and they prefer to label a lot of examples to always have a known template to fall back on.

If you are in that case, you can speed up your test execution by running template match first before the regular ML.

To do this, pass in do_exact_match_first=True to the initialization of the driver. Furthermore, you can configure your own threshold for the template matching by passing in exact_match_first_threshold to the initialization of the driver.

smart_driver = SmartDriver(driver, "??API_KEY??", initialization_dict={"do_exact_match_first": True, "exact_match_first_threshold": 0.99})

Configure timeouts​

The SDK has some built in timeouts to make sure that the tests don't hang forever. If you see timeouts regularly, you can increase the timeouts by passing in the following parameters to the initialization of the driver.

There are two timeouts you can configure:

  • detect_timeout - The time to wait in seconds for the element to be classified by the server.
  • misc_timeout - The time out in seconds for all the other operations, including screenshot upload.

The timeout is the base value, retries up to 3 times are implemented where the timeout is doubled each time from the base value.

smart_driver = SmartDriver(driver, "??API_KEY??", initialization_dict={"ai_timeout": 10, "template_match_timeout": 10})

Retries​

The SDK will retry to find the element 3 times by default (only in Java). You can configure the number of retries with classifyMaxRetries

not_implemented

Direct element with Chrome Developer Protocol - Flutter - use cdp​

Some apps are hard to automate because they do pure javascript rendering without a DOM tree. Some other time there is no good HTML element under the element you want to click on, like games, maps and Flutter apps. In situations like this, we can entirely work based on pixel coordinates on screen for that element, we call it the CDP mode.

This feature is only available in the Python SDK for now. Let us know if you need it in Java.

To enable it, pass in use_cdp=True to the initialization of the driver.

smart_driver = SmartDriver(driver, "??API_KEY??", initialization_dict={"use_cdp": True})

Local template match mode - local_caching​

If you are want to speed up your test runs, you can try local caching. This will run template match locally on your elements and only call out to the web backend when failing. If the flexibility of AI matching is not really required, feel free to use this.

To enable it, pass in local_caching=True to the initialization of the driver. Additionally you can configure the template match threshold with local_match_threshold

This feature is only available in the Python SDK for now. Let us know if you need it in Java.

smart_driver = SmartDriver(driver, "??API_KEY??", initialization_dict={"local_caching": True, "local_match_threshold": 0.99})