Skip to content

Open In Colab

How to import assets to a Kili project

In this tutorial, we will learn how to import assets to your project, and add metadata to those assets.

Here are the steps that we will follow:

  1. Setting up a simple Kili project to work with
  2. Importing assets to Kili
  3. Adding metadata to assets

Setting up a simple Kili project to work with

Installing and instantiating Kili

First, let's install and import the required modules.

!pip install  kili

import getpass
import os

from kili.client import Kili

Now, let's set up variables needed to create an instance of the Kili object.

We will need your API key and Kili's API endpoint.

If you are unsure how to look up your API key, refer to https://docs.kili-technology.com/docs/creating-an-api-key.

if "KILI_API_KEY" not in os.environ:
    KILI_API_KEY = getpass.getpass("Please enter your API key: ")
else:
    KILI_API_KEY = os.environ["KILI_API_KEY"]

With variables set up, we can now create an instance of the Kili object.

kili = Kili(
    api_key=KILI_API_KEY,  # no need to pass the API_KEY if it is already in your environment variables
    # api_endpoint="https://cloud.kili-technology.com/api/label/v2/graphql",
    # the line above can be uncommented and changed if you are working with an on-premise version of Kili
)

Creating a basic Kili project

To create a Kili project, you must first set up its interface.

We will create a simple image project with just one simple classification job and two categories: OBJECT_A and OBJECT_B.

To learn more about Kili project interfaces, refer to https://docs.kili-technology.com/docs/customizing-project-interface.

interface = {
    "jobs": {
        "JOB_0": {
            "mlTask": "CLASSIFICATION",
            "required": 1,
            "content": {
                "categories": {"OBJECT_A": {"name": "Object A"}, "OBJECT_B": {"name": "Object B"}},
                "input": "radio",
            },
        }
    }
}

result = kili.create_project(
    title="Test Project",
    description="Project Description",
    input_type="IMAGE",
    json_interface=interface,
)

For further processing, we will need to find out what our project ID is.

We can easily retrieve it from the project creation response message:

project_id = result["id"]
print("Project ID: ", project_id)

Project ID:  clfwarjr500110jqbdgnw14kv

Importing assets to Kili

Now, let's add some assets to be labeled.

We will use some free off-the-shelf examples from the Internet.

url1 = "https://storage.googleapis.com/label-public-staging/car/car_2.jpg"
url2 = "https://storage.googleapis.com/label-public-staging/car/car_1.jpg"
url3 = "https://storage.googleapis.com/label-public-staging/recipes/inference/black_car.jpg"

assets = kili.append_many_to_dataset(
    project_id=project_id,
    content_array=[url1, url2, url3],
    external_id_array=["image_1", "image_2", "image_3"],  # name to give to assets
)

At this point, you should be able to see your assets in your Kili project:

image.png

If you prefer to add your own images, you can use a local file. The code to do that would look similar to this:

project_id = 'project_id'
assets = kili.append_many_to_dataset(
    project_id=project_id,
    content_array=['./image_1.jpeg'], # Path to local image
    external_id_array=['image_1']
)

The procedure looks the same for most of other data types, like PDFs or text. For more information on supported file formats, refer to our documentation.

Because videos and Rich Text assets may be more complex to import, we've created separate tutorials devoted to them:

  • For information on importing video assets, refer to this tutorial.
  • For information on importing Rich Text assets, see here.

For more information on importing assets, refer to our documentation.

Adding metadata to assets

In Kili, you can add extra information to an asset by using asset metadata. The metadata can contain extra information like what language the document was written in, custom quality metrics, agreement metrics etc., and can be used with Kili's advanced filters. The metadata can also contain text extracted from images or PDF documents using OCR (Optical Character Recognition), and that will be shown in the Kili labeling interface.

Additionally, three specific metadata types can be used as information presented to labelers in Kili interface:

  • imageUrl
  • text
  • url

As an optional step, you can set data types for each type of your metadata. The default data type is string, but setting some of your metadata as number can really help apply filters on your assets later on.

Note that we don't need to set data types for imageUrl, text, and url.

kili.update_properties_in_project(
    project_id=project_id,
    metadata_types={
        "customConsensus": "number",
        "sensitiveData": "string",
        "uploadedFromCloud": "string",
        "modelLabelErrorScore": "number",
    },
)

{'id': 'clfwarjr500110jqbdgnw14kv',
 'metadataTypes': {'sensitiveData': 'string',
  'customConsensus': 'number',
  'uploadedFromCloud': 'string',
  'modelLabelErrorScore': 'number'}}

Now we can add metadata to our assets:

external_ids = ["image_1", "image_2"]

kili.update_properties_in_assets(
    project_id=project_id,
    external_ids=external_ids,
    json_metadatas=[
        {
            "customConsensus": 10,
            "sensitiveData": "yes",
            "uploadedFromCloud": "no",
            "modelLabelErrorScore": 50,
        },
        {
            "customConsensus": 40,
            "sensitiveData": "no",
            "uploadedFromCloud": "yes",
            "modelLabelErrorScore": 30,
        },
    ],
)

# Add metadata that will be visible to labelers in the labeling interface:
kili.update_properties_in_assets(
    project_id=project_id,
    external_ids=external_ids,
    json_metadatas=[
        {"imageUrl": "www.example.com/image.png", "text": "some text", "url": "www.example.com"},
        {"imageUrl": "www.example.com/image.png", "text": "some text", "url": "www.example.com"},
    ],
)

If you want to add metadata based on Optical Character Recognition, the process is slightly different. To help you with it, we've created a separate tutorial.

For more information on adding asset metadata, refer to our documentation.

Cleanup

We can remove the project that we created:

kili.delete_project(project_id);

Summary

We've successfully set up a Kili project, imported assets to it, and finally added some metadata to our assets. Well done!