How to develop and test a Kili plugin

Preliminary

This notebook will teach you how to build your first plugin.

A plugin is an uploaded Python script triggered by an event that you define.

For instance, you can trigger a plugin when a labeler clicks on Submit with the on_submit handler.

The plugin should have different methods for the different types of events:

on_submit
on_review

These methods have a predefined set of parameters:

the label submitted
the asset_id of the asset labeled

Some attributes are available in the class:

self.kili
self.project_id

Therefore, the skeleton of the plugin should look like this:

from kili.plugins import PluginCore
from typing import Dict
import numpy as np

class PluginHandler(PluginCore):
    """Custom plugin"""

    def on_review(self, label: Dict, asset_id: str) -> None:
        """Dedicated handler for Review action"""
        # Do something...

    def on_submit(self, label: Dict, asset_id: str) -> None:
        """Dedicated handler for Submit action"""
        # Do something...

Do not hesitate to reach out to us if you need more.

NB: The plugin capabilities of Kili are under active development, and compatible with version 2.125.2 and later of Kili. Don't hesitate to reach out via Github or Kili support to provide feedback.

Instantiate Kili

!pip install  kili

%load_ext autoreload
%autoreload 2

import os

from kili.client import Kili

api_endpoint = os.getenv("KILI_API_ENDPOINT")
api_key = os.getenv("KILI_API_KEY")


kili = Kili(api_endpoint=api_endpoint, api_key=api_key)

Develop your plugin

The first step is to define the functions that will be called when the event is triggered. You will be able to iterate on these functions locally (more on that in the next section).

The plugin can be defined in two ways: a single .py file with everything inside or a module (folder containing multiple .py files). In the case of the module type, a file named main.py needs to be at the root of the folder and will serve as the entrypoint.

1. First option - Plugin defined in a single file

This cell should be the contents of the .py file that you will upload as a plugin at the end.

This file should define the PluginHandler class that will contain the proper methods.

We recommend using a modern IDE like VScode to get type hints and autocompletion on the methods.

from typing import Dict

import numpy as np

from kili.plugins import PluginCore


def custom_function(label: Dict):
    label_id = label.get("id")
    print(f"My custom function for review of label with id {label_id}")


class PluginHandler(PluginCore):
    """
    Custom plugin instance
    """

    def custom_method(self, project_id, label_id):
        print(f"custom_method called for label {label_id}")
        random_seed = np.random.random(1)[0]
        if random_seed > 0.5:
            self.logger.warning("Generating issue")
            # Use kili for actions with self.kili
            self.kili.create_issues(
                project_id=project_id,
                label_id_array=[label_id],
                text_array=["Random issue generated for this label"],
            )

    def on_review(self, label: Dict, asset_id: str) -> None:
        """
        Dedicated handler for Review action
        """
        custom_function(label)

    def on_submit(self, label: Dict, asset_id: str) -> None:
        """
        Dedicated handler for Submit action
        """
        print("On submit called")

        project_id = self.project_id
        label_id = label.get("id")

        self.custom_method(project_id, label_id)

2. Second option - Plugin defined in a folder

As said previously, the structure of the folder can be the following (the only constraint being the presence of the main.py file):

plugin_folder
|__ main.py
|__ other_file.py
|__ requirements.txt
|
|___helpers
    |__ helper.py

You can notice that you can also include a requirements.txt file in the folder and the necessary packages will be installed with your plugin. Don't forget to add them, since the plugin could work on your machine if you have them installed, but it won't be possible to create the plugin if there are missing dependencies.

Important: The main.py file need to have the same skeleton as the plugin defined in a single file (presence of the class PluginHandler), the difference being that it can import and call functions defined in other files

Depending on where the folder is stored, there are two ways to import the plugin in order to test it:

The first way is to use a relative import (having the plugin folder and the notebook in the same folder). It is simpler and we recommend it as it will also allow the IDE to detect the correct methods and propose hints and autocompletion.
The second is to use an absolute path to the plugin folder

2.1 Relative import

# Here replace 'plugin_folder' with the actual name of the folder
from plugin_folder.main import PluginHandler

2.2 Absolute path import

import os
import sys
from pathlib import Path

# Input the path to the plugin folder (it should include the folder), for example '/path/to/plugin_folder'
plugin_path = "<INSERT PATH TO PLUGIN FOLDER>"

module_path = str(Path(plugin_path).parent.absolute())

# We are inserting the path in the system PATH to be able to import the module in the next line
sys.path.insert(0, module_path)

# In the next line replace 'plugin_folder' with the actual name of the folder
from plugin_folder.main import PluginHandler

Testing the plugin locally

In this we will show you how to test your plugin locally before uploading it.

project_id = "<PROJECT ID>"

Instantiate the plugin:

my_plugin_instance = PluginHandler(kili, project_id)


def get_label(label_id, project_id):
    """
    Function to get the object Label with the same keys as it will be in the plugin
    """
    label = list(
        kili.labels(
            project_id=project_id,
            label_id=label_id,
            fields=["id", "jsonResponse", "author.id", "labelType", "createdAt", "secondsToLabel"],
        )
    )[0]

    label["authorId"] = label["author"]["id"]
    del label["author"]
    return label

Test the plugin run

If you already have a test project with labels added, you can directly use the IDs of these labels (see the following cell). Otherwise, you can follow the plugins_example.ipynb notebook to create a new project and then upload an asset with an associated label.

asset_id = "<YOUR_ASSET_ID>"
label_id = "<YOUR_LABEL_ID>"

label = get_label(label_id=label_id, project_id=project_id)

my_plugin_instance.on_submit(label=label, asset_id=asset_id)

Test the plugin run on Kili

When you finish debugging the code, you may want to upload it directly into Kili.

Note that you might get an error if the plugin name already exists in your Kili organization.

path_to_plugin = "path/to/my/plugin.py"
plugin_name = "My first kili plugin"

from kili.exceptions import GraphQLError

try:
    kili.upload_plugin(path_to_plugin, plugin_name)
except GraphQLError as error:
    print(str(error))

Plugins must be activated in the project that you want them to run in. Be careful with production projects: your custom workflows or rules will also be applied

kili.activate_plugin_on_project(plugin_name, project_id=project_id)

Monitoring the plugin

Plugin creation takes some time (around 5 minutes). The plugin will begin to run only after it's been fully created (if labeling events are to be triggered on this project).

Additionally, you can get the logs of the runs:

kili.get_plugin_logs(project_id=project_id, plugin_name=plugin_name)

You can set custom date rules for filtering your logs:

from datetime import date, datetime

dt = date.today()  # You can change this date if needed
start_date = datetime.combine(dt, datetime.min.time())

kili.get_plugin_logs(project_id=project_id, plugin_name=plugin_name, start_date=start_date)

Managing your plugin

There are several other methods to manage your plugins and their lifecycle. To find out more, check the plugins tutorials.