Skip to content

Open In Colab

Plugins Example

Context

This notebook is an end-to-end example that you can follow to: create a project, upload a first plugin and activate it on this project, and finally start monitoring it.

NB: The plugin capabilities of Kili are under active development, and compatible with version 2.125.2 and later of Kili. Don't hesitate to reach out via Github or the Kili support to provide feedback.

Step 1: Instantiate Kili

!pip install --upgrade kili
from kili.client import Kili
import os
import time
from pathlib import Path

kili = Kili()

Step 2: Create the project

First, we need to create a new project. In our example, we will use an IMAGE type project with the following jsonInterace:

json_interface = {
    "jobs": {
        "JOB_0": {
            "content": {
                "categories": {
                    "OBJECT_A": {
                        "children": [],
                        "name": "Object A",
                        "color": "#733AFB",
                        "id": "category1",
                    },
                    "OBJECT_B": {
                        "children": [],
                        "name": "Object B",
                        "color": "#3CD876",
                        "id": "category2",
                    },
                },
                "input": "radio",
            },
            "instruction": "Categories",
            "isChild": False,
            "tools": ["rectangle"],
            "mlTask": "OBJECT_DETECTION",
            "models": {},
            "isVisible": True,
            "required": 1,
            "isNew": False,
        }
    }
}
title = "Plugins test project"
description = "My first project with a plugin"
input_type = "IMAGE"

project = kili.create_project(
    title=title, description=description, input_type=input_type, json_interface=json_interface
)
project_id = project["id"]

print(f"Created project {project_id}")
Created project clcysbp9o2d7w0krfghuf7le1

Upload an asset:

content_array = ["https://storage.googleapis.com/label-public-staging/car/car_1.jpg"]
names_array = ["landscape"]

kili.append_many_to_dataset(
    project_id=project_id, content_array=content_array, external_id_array=names_array
)

asset_id = list(kili.assets(project_id=project_id, fields=["id"], disable_tqdm=True))[0]["id"]

This project has one job of bounding box creation with two categories.

With our plugin, we want to make sure that the labelers don't create more than one bounding box of category A.

To iterate on the plugin code, you can refer to the plugins_development.ipynb notebook.

Step 3: Write the plugin

from kili.plugins import PluginCore
from kili.types import Label

def check_rules_on_label(label: Label):
    #custom methods
    print('Custom method - checking number of bboxes')

    counter = 0
    for annotation in label['jsonResponse']["JOB_0"]["annotations"]:
        if annotation["categories"][0]["name"] == "OBJECT_A":
            counter += 1

    if counter == 0:
        return []
    return [f'There are too many BBox ({counter}) - Only 1 BBox of Object A accepted']

class PluginHandler(PluginCore):
    """
    Custom plugin instance
    """

    def on_submit(self, label: Label, asset_id: str) -> None:
        """
        Dedicated handler for Submit action
        """
        self.logger.info("On submit called")

        issues_array = check_rules_on_label(label)

        project_id = self.project_id

        if len(issues_array) > 0:
            print("Creating an issue...")

            for i, _ in enumerate(issues_array):

                self.kili.append_to_issues(
                    label_id=label['id'],
                    project_id=project_id,
                    text=issues_array[i],
                )

            print("Issue created!")

            self.kili.send_back_to_queue(asset_ids=[asset_id])
import urllib.request


plugin_folder = "plugin_folder"

os.mkdir(plugin_folder)
urllib.request.urlretrieve(
    "https://raw.githubusercontent.com/kili-technology/kili-python-sdk/master/recipes/datasets/plugins/plugin.py",
    "plugin_folder/main.py",
);

Step 4: Upload the plugin from a folder

With the plugin defined in a separate Python file, you can create a folder containing: - A main.py file which is the entrypoint of the plugin and must have a PluginHandler class which implements a PluginCore class - (optionally) a requirements.txt (if you need specific PyPi packages in your plugin)

folder/
     main.py
     requirements.txt
  • The upload will create the necessary builds to execute the plugin (it will take a few minutes)
  • After the activation, you can start using your plugin right away.

Here is an example of a requirements.txt file: numpy scikit-learn pandas==1.5.1 git+https://github.com/yzhao062/pyod.git

requirements_path = Path(plugin_folder) / "requirements.txt"

packages_list = [
    "numpy\n",
    "scikit-learn\n",
    "pandas==1.5.1\n",
    "git+https://github.com/yzhao062/pyod.git\n",
]

with requirements_path.open("w") as f:
    f.writelines(packages_list)
plugin_name = "Plugin bbox count"

from kili.exceptions import GraphQLError

try:
    kili.upload_plugin(plugin_folder, plugin_name)
except GraphQLError as error:
    print(str(error))
Hint: A plugin with this name already exist, if you want to override it you can use the command kili.update_plugin(plugin_path="plugin.py", plugin_name="Plugin bbox count")
error: "[pluginsError] An error occured handling your plugin -- This can be due to: 400: Bad Request: createPlugin: an entity Plugin already exists with value "Plugin bbox count" for field 'name' | trace : false"
kili.activate_plugin_on_project(plugin_name, project_id=project_id);
Plugin with name "Plugin bbox count" activated on project "clcysbp9o2d7w0krfghuf7le1"

Step 4 bis: Upload the plugin from a .py file

Alternatively, you can also create a plugin directly from a .py file.

  • The upload will create the necessary builds to execute the plugin (it will take a few minutes)
  • After the activation, you can start using your plugin right away.
path_to_plugin = Path(plugin_folder) / "main.py"
plugin_name = "Plugin bbox count"

from kili.exceptions import GraphQLError

try:
    kili.upload_plugin(str(path_to_plugin), plugin_name)
except GraphQLError as error:
    print(str(error))
kili.activate_plugin_on_project(plugin_name, project_id=project_id);

Step 5: Plugin in action

Wait for the plugin to be successfully deployed.

After that, you can test it by labelling in the Kili interface or just by uploading the following label.

When you add the label that contains errors, you will see a new issue automatically created in the Kili app.

json_response = {
    "JOB_0": {
        "annotations": [
            {
                "boundingPoly": [
                    {
                        "normalizedVertices": [
                            {"x": 0.15, "y": 0.84},
                            {"x": 0.15, "y": 0.31},
                            {"x": 0.82, "y": 0.31},
                            {"x": 0.82, "y": 0.84},
                        ]
                    }
                ],
                "categories": [{"name": "OBJECT_A"}],
                "children": {},
                "mid": "20221124161451411-13314",
                "type": "rectangle",
            },
            {
                "boundingPoly": [
                    {
                        "normalizedVertices": [
                            {"x": 0.79, "y": 0.20},
                            {"x": 0.79, "y": 0.13},
                            {"x": 0.91, "y": 0.13},
                            {"x": 0.91, "y": 0.20},
                        ]
                    }
                ],
                "categories": [{"name": "OBJECT_A"}],
                "children": {},
                "mid": "20221124161456406-47055",
                "type": "rectangle",
            },
            {
                "boundingPoly": [
                    {
                        "normalizedVertices": [
                            {"x": 0.87, "y": 0.36},
                            {"x": 0.87, "y": 0.27},
                            {"x": 0.99, "y": 0.27},
                            {"x": 0.99, "y": 0.36},
                        ]
                    }
                ],
                "categories": [{"name": "OBJECT_A"}],
                "children": {},
                "mid": "20221124161459298-45160",
                "type": "rectangle",
            },
        ]
    }
}
kili.append_labels(
    json_response_array=[json_response], asset_id_array=[asset_id], label_type="DEFAULT"
)
[{'id': 'clcysc83n2asf0lq483rlc5wv'}]

If you use the base plugin provided, the plugin should:

  • Create an issue with information that three bboxes were found, instead of one
  • Send the asset back to the labeling queue (status ONGOING)
print(
    kili.assets(project_id=project_id, asset_id=asset_id, fields=["status", "issues.comments.text"])
)

print(
    f'Go to my project: {kili.auth.api_endpoint.split("/api")[0]}/label/projects/{project_id}/menu/queue'
)
[{'issues': [], 'status': 'LABELED'}]
Go to my project: https://cloud.kili-technology.com/label/projects/clcysbp9o2d7w0krfghuf7le1/menu/queue

Woah! Amazing! Well done :) 🚀

Let's test now to post a proper label, this one for example:

json_response = {
    "JOB_0": {
        "annotations": [
            {
                "boundingPoly": [
                    {
                        "normalizedVertices": [
                            {"x": 0.15, "y": 0.84},
                            {"x": 0.15, "y": 0.31},
                            {"x": 0.82, "y": 0.31},
                            {"x": 0.82, "y": 0.84},
                        ]
                    }
                ],
                "categories": [{"name": "OBJECT_A"}],
                "children": {},
                "mid": "20221124161451411-13314",
                "type": "rectangle",
            }
        ]
    }
}
kili.append_labels(
    json_response_array=[json_response], asset_id_array=[asset_id], label_type="DEFAULT"
)

print(kili.assets(project_id=project_id, asset_id=asset_id, fields=["status"]))

print(
    f'Go to my project: {kili.auth.api_endpoint.split("/api")[0]}/label/projects/{project_id}/menu/queue'
)
[{'status': 'LABELED'}]
Go to my project: https://cloud.kili-technology.com/label/projects/clcysbp9o2d7w0krfghuf7le1/menu/queue

The status of your asset should have now changed to LABELED. In this plugin, previous issues remain but you can solve them through the API as well.

Well done! You can now iterate on the script. To learn how to avoid latency when building and deploying your plugin, refer to the plugins_development.ipynb tutorial.

Step 6: Monitor the plugin

To monitor a certain plugin, you can get its logs by using the following command:

import json
from datetime import date
from datetime import datetime

dt = (
    date.today()
)  # You can change this date if needed, or omit it to set it at the plugin creation date
start_date = datetime.combine(dt, datetime.min.time())

logs = kili.get_plugin_logs(project_id=project_id, plugin_name=plugin_name, start_date=start_date)

logs_json = json.loads(logs)
print(json.dumps(logs_json, indent=4))
[
    {
        "content": "Issue created!",
        "createdAt": "2023-01-16T12:31:27.633Z",
        "logType": "info",
        "metadata": {
            "assetId": "clcysbre30000jevzat4b7k4d",
            "labelId": "clcysc83n2asf0lq483rlc5wv"
        },
        "pluginName": "Plugin bbox count",
        "projectId": "clcysbp9o2d7w0krfghuf7le1",
        "runId": "clcysc93e07wv014z5nbc0kfc"
    },
    {
        "content": "Creating an issue...",
        "createdAt": "2023-01-16T12:31:26.620Z",
        "logType": "info",
        "metadata": {
            "assetId": "clcysbre30000jevzat4b7k4d",
            "labelId": "clcysc83n2asf0lq483rlc5wv"
        },
        "pluginName": "Plugin bbox count",
        "projectId": "clcysbp9o2d7w0krfghuf7le1",
        "runId": "clcysc93e07wv014z5nbc0kfc"
    },
    {
        "content": "Custom method - checking number of bboxes",
        "createdAt": "2023-01-16T12:31:26.619Z",
        "logType": "info",
        "metadata": {
            "assetId": "clcysbre30000jevzat4b7k4d",
            "labelId": "clcysc83n2asf0lq483rlc5wv"
        },
        "pluginName": "Plugin bbox count",
        "projectId": "clcysbp9o2d7w0krfghuf7le1",
        "runId": "clcysc93e07wv014z5nbc0kfc"
    },
    {
        "content": "On submit called",
        "createdAt": "2023-01-16T12:31:26.619Z",
        "logType": "info",
        "metadata": {
            "assetId": "clcysbre30000jevzat4b7k4d",
            "labelId": "clcysc83n2asf0lq483rlc5wv"
        },
        "pluginName": "Plugin bbox count",
        "projectId": "clcysbp9o2d7w0krfghuf7le1",
        "runId": "clcysc93e07wv014z5nbc0kfc"
    }
]

Step 7: Manage the plugin

You also have several other methods to manage your plugins.

Get the list of all uploaded plugins in your organization:

plugins = kili.list_plugins()

Update a plugin with new source code:

new_path_to_plugin = "plugin.py"
if new_path_to_plugin != path_to_plugin:
    kili.update_plugin(plugin_name=plugin_name, plugin_path=new_path_to_plugin)

Deactivate the plugin on a certain project (the plugin can still be active for other projects):

kili.deactivate_plugin_on_project(plugin_name=plugin_name, project_id=project_id);
Plugin Plugin bbox count deactivated on project clcysbp9o2d7w0krfghuf7le1

Delete the plugin completely (deactivates the plugin from all projects):

if delete_plugin_from_org:
    kili.delete_plugin(plugin_name=plugin_name)