Welcome to Pybot’s documentation!

Indices and tables

• Index
• Module Index
• Search Page

About Pybot

Pybot

This automation framework aims to interact with GUI (web, desktop, android, etc) from the computer, or a connected Android. Many tasks could be automated, like information harvesting or tests. Script can be designed with Sikuli IDE. The framework then convert Sikuli project to python 3 to take advantage of that powerful scripting language and the python 3 ecosystem. Computer vision, mouse event and keyboard event management are perfect for workstation testing and automation. For Android only, Appium or similar technologies not using computer vision are more reliable. If Appium can’t be used, or for any GUI in a VM without accessible tags, computer vision based technologies are still a good option for testing and automation.

Platforms

At the moment, for computer, only Windows platforms are supported. Android can be accessed by mirroring the phone screen thanks to scrpy embedded in this framework.

Why another framework

Sikuli IDE is an extremely powerful tool to script automation through any GUI, desktop application, web browser, android, etc… The only python3 compliant wrapper around Sikuli is lackey. This then use lackey and other libraries to complete the laking features of Sikuli. Unlike pure Sikuli that use JVM and therefore Jython to access some low level API. This framework access platform API via robust python methods. For instance, using real python allow us to pytest. Other methods are available from the Pybot class to easily extract information from the GUI like text by used Tesseract OCR.

Dependencies

Python 3.6 with virtualenv to create the virtual env. The all purpose of this framework is to use to possibility to script GUI elements with Sikuli IDE to initiate scripts and class. Sikuli is java based, therefore you need Java. To analyse image text, this framework needs Tesseract OCR. For the python wrapper to work the tesseract command has to be in the path. It should be something like C:\Program Files (x86)\Tesseract-OCR.

Pip packages

The pip package lackey (the only python3 Sikuli wrapper) , virtualenv, pytest, pytest-html, doxypypy, lackey, wheel, easygui and pytesseract pip packages. Sikuli method are named like built-in or classical python method like type(). lackey renamed the original method an _, type() becomes type_(). For the same reason it also depend on the pip package virtualenv, cause it is highly recommended to run this library out of system python interpreter.

CLI

Download from this git repository and create the virtulenv:

git clone https://github.com/phe-sto/Pybot.git
cd Pybot
Pybot setup

Export Sikuli class from your sikuli project in the sikuli_project/ folder:

Pybot export class <your project name> Export Sikuli

script from your sikuli project in the sikuli_project/ folder:

Pybot export script <your project name>

Scripts and classes are available in the venv virtualenv. To activate this one:

Pybot activtate

Clear the virtualenv:

Pybot clear

API

Full documentation API can be found here.

Pybot API

Pybot

Pybot class can be used as Jython source to be used in sikuli IDE, not pure python or python3. Initialize the script by importing the required libraries for test (a.k.a unittest), subprocess and system to using Windows commands. Lackey package provide a wrapper arround Sikuli. Contain sources description. Custom class to be used in as an abstraction layer to sikuli methods and corresponding exception. Can used command to start/stop program or use commands, access Android mirroring. Interpretation of GUI, access to windows system commands.

Warning

Coding style is CamelCase for classes and lowercase_separated_by_underscores A.K.A snake_case for methods and variables.

class Pybot.Pybot.Pybot(cache=True)

Something to automate on a computer a task, a test, etc…”

android()

Access connected Android device via adb.exe.

return:Tuple containing the Android Serial number and device type.

android_app_switch()

click on APP_SWITCH.

android_back()

click on BACK.

android_connected()

Check if an Android is connected.

return:True if command Android connected, False on contrary.

android_fullscreen()

Type the scrcpy shortcut to full screen f + CTRL.

android_home()

click on HOME.

android_number()

Calculate the number of Android device connected.

return:Integer corresponding to the number of Android device(s).

android_paste_clipboard()

paste computer clipboard to device2

android_power()

click on POWER.

android_remove_black_borders()

resize window to remove black borders.

android_resize_one_to_one()

resize window to 1:1 (pixel-perfect).

android_toggle_fps_counter()

enable/disable FPS counter (on stdout)2

android_volume_down()

click on VOLUME_DOWN.

android_volume_up()

click on VOLUME_UP.

check_android_gui()

Check that Android mirroring with SCRCPY_EXE is running.

return:Boolean True if mirroring, False on contrary.

check_click(img, sleep_sec=0, after_click=None)

Method checking if button exist and clicking on it, return True is clicked False on contrary. Eventually sleep.

param img:Image path to work on. Check if exist and click. param sleep_sec:  Number of seconds of seconds to eventually sleep after the click. param after_click:  Another image to eventually click after the first click and before the sleep. return:True if img was found and clicked, False on contrary. raise TypeError:  If arg (img) or kwarg (after_click) are not string type. raise PybotException:  If img or after_click file path does not exist.

check_pgm(pgm)

Check if a program is running.

param pgm:

Program to check, with the .exe extension.

return:

True if program running, false on contrary.

raise TypeError:  

If only argument pgm is not a string.

examples:

test_automaton = Pybot()
test_automaton.check_pgm("Firefox.exe")

ctrl_shorcut(key)

Type a key with the key modifier CTRL.

param key:The key to type with the CTRL modifier.

exec_cmd(cmd, sleep_sec=0, cwd='./')

Execute command on Windows OS.

param cmd:

Command to execute passed a string.

param sleep_sec:  

Number of seconds to eventually sleep after the click.

return:

True if return code of the command is 0, false on contrary.

raise TypeError:  

If first argument cmd is not a string type.

examples:

test_automaton = Pybot()
test_automaton.exec_cmd("DIR")

export_sikuli_class(project_name)

Export a sikuli project class to the Pybot package on Windows OS.

param project_name:  

Name of the project to export.

return:

True if class file created, False on contrary.

examples:

test_automaton = Pybot()
test_automaton.export_sikuli_class("tahomaBee")

export_sikuli_script(project_name)

Export a sikuli project script to the script library ./ on Windows OS.

param project_name:  

Name of the project to export.

return:

True if script file created, False on contrary.

examples:

test_automaton = Pybot()
test_automaton.export_sikuli_script("tahomaBee")

get_text_img(img_file, lang=None)

Retrieve text from an image.

param img_file:

Name of the image file.

param lang:

None is default, this parameter specify a language to tesseract.

return:

String of the text decrypted

raise TypeError:  

If wrong bounds kwarg type.

raise PybotException:  

If wrong tesseract lang kwarg (tesseract language). Default is None.

examples:

test_automaton = Pybot()
test_automaton.screenshot("1234567891012.png",lang='eng')

kill_pgm(pgm, sleep_sec=0)

Kill a program

param pgm:Program to kill, with the .exe extension. param sleep_sec:  Number of seconds of seconds to eventually sleep after the click. return:True if return code of the command is 0, false on contrary. raise TypeError:  If kwarg pgm_arg or working_direcory kwarg are not None or string type.

purge_cache()

Deleting cache database.

return:True if cache is clear, False on contrary.

screenshot(bounds=None, text=False, lang=None)

Taking a screenshot, default is all the screen.

param bounds:

The bounds of the image to take, default is None, to get the all screen.

param text:

Boolean True to discover text, False on contrary.

param lang:

Specify a lang for the image text. Default is None.

return:

A tuple made of the boolean integer, image file and thetext discovered in the image.

raise TypeError:  

If wrong bounds kwarg type.

raise PybotException:  

If wrong tesseract lang kwarg (tesseract language).

example:

test_automaton = Pybot()
test_automaton.screenshot(lang='eng') # Screenshot of the full screen with english text description

start_android_gui(sleep_sec=5, fullscreen=True)

Start Android mirroring with SCRCPY_EXE if connected and full screen it.

param sleep_sec:  Number of seconds to eventually sleep after the click. param fullscreen:  If True, start android scrcpy in fullscreen. raise TypeError:  If kwarg fullscreen is not an boolean type. return:Boolean True if started, False on contrary.

start_pgm(pgm, working_directory=None, pgm_arg=None, sleep_sec=0)

Start a program in background in a given directory on Windows OS.

param pgm:

Program to start.

param working_directory:  

Working directory to start the program, with the .exe extension.

param pgm_arg:

Eventual argument of the program to start.

param sleep_sec:  

Number of seconds of seconds to eventually sleep after the click.

return:

True if return code of the command is 0, false on contrary.

raise TypeError:  

If kwarg pgm_arg or working_direcory kwarg are not None or string type.

examples:

test_automaton = Pybot()
test_automaton.start_pgm('node.exe', working_directory='server', sleep_sec=5)

start_web(url, sleep_sec=0)

Start a website on the default browser, wait 5 seconds for it to open and full screen it on Windows OS.

param url:

URL of the website.

param sleep_sec:  

Number of seconds of seconds to eventually sleep after the click.

return:

True if return code of the command is 0, false on contrary.

raise TypeError:  

If only argument url is not a string.

examples:

test_automaton = Pybot()
test_automaton.start_web("https://papit.fr")

stop_android_gui()

Stop SCRCPY_EXE processes.

return:Boolean True if command executed correctly, False on contrary.

text(bounds=None, lang=None)

Retrieve the text on the screen, default is all the screen.

param bounds:

The bounds of the image to take, default is None, to get the all screen.

param lang:

Specify a lang for the image text by tesseract.

return:

The string decrypted from the screen.

raise TypeError:  

If wrong bounds kwarg type. Default is None.

raise PybotException:  

If wrong tesseract lang kwarg (tesseract language). Default is None.

example:

test_automaton = Pybot()
test_automaton.text(lang='eng') # get text of the full screen with english text description

turn_screen_on()

turn screen on.

type_n_time(n, key, sleep_sec=0)

Type n time the desired key. Eventually sleep sleep_sec seconds.

param n:

Number of time to type the key.

param key:

Key to type.

param sleep_sec:  

Number of seconds of seconds to eventually sleep after the click.

raise TypeError:  

n must be an integer or float type.

examples:

test_automaton = Pybot()
test_automaton.type_n_time(5, Key.11)

wait_click(img, sleep_sec=0)

Method that wait for a button to appear and click on it. Eventually sleep sleep_sec seconds after.

param img:Image to wait for and click. param sleep_sec:  Number of seconds of seconds to eventually sleep after the click. raise TypeError:  If first argument img is not string type. raise PybotException:  If first argument img file path does not exist.

exception Pybot.Pybot.PybotException

Automation exception