How to work with audio projects in Kili

Kili audio projects are designed for speech transcription with speaker attribution: a labeler listens to a recording, draws segments on the waveform, types what is being said in each of them, and assigns each segment to a speaker.

In this tutorial, we will go through the full life cycle of an audio project:

Setting up an audio project
Importing audio assets
Understanding the audio label format
Importing model pre-annotations (predictions)
Exporting audio labels
Cleanup

Let's start by installing the SDK and instantiating the client. Kili() reads your API key from the KILI_API_KEY environment variable — see how to create one.

%pip install  kili

import json

from kili.client import Kili

kili = Kili()

1. Setting up an audio project

Designing the labeling interface

An audio interface is made of two very different kinds of jobs, and it is worth understanding the distinction before writing any code.

Segment-level jobs. A TRANSCRIPTION job that is not flagged as asset-level is the transcription job of the project. It is the job that materializes the segments drawn on the waveform: every segment is one annotation of that job, with a time interval, a piece of text and a speaker. An audio project has exactly one such job — it is what makes the waveform editable.

Asset-level jobs. Any job carrying "level": "asset" applies to the whole recording rather than to a segment. They are rendered in a side panel on the right of the interface and behave like the classification and transcription jobs you already know from image or text projects. Use them for metadata such as the language of the call, the audio quality, or a free-text summary.

Let's build an interface with one transcription job and two asset-level classification jobs.

json_interface = {
    "jobs": {
        # Segment-level job: this is the job the waveform segments belong to.
        # A TRANSCRIPTION job without "level": "asset" is the transcription job of the project.
        "TRANSCRIPTION_JOB": {
            "mlTask": "TRANSCRIPTION",
            "content": {"input": "textField"},
            "instruction": "Transcription",
            "required": 0,
            "isChild": False,
        },
        # Asset-level job: applies to the whole recording, shown in the right-hand panel.
        "LANGUAGE_JOB": {
            "mlTask": "CLASSIFICATION",
            "content": {
                "categories": {
                    "ENGLISH": {"children": [], "name": "English", "id": "category_english"},
                    "FRENCH": {"children": [], "name": "French", "id": "category_french"},
                    "OTHER": {"children": [], "name": "Other", "id": "category_other"},
                },
                "input": "singleDropdown",
            },
            "instruction": "Language of the recording",
            "required": 1,
            "isChild": False,
            "level": "asset",
        },
        "AUDIO_CHARACTERISTICS_JOB": {
            "mlTask": "CLASSIFICATION",
            "content": {
                "categories": {
                    "BACKGROUND_MUSIC": {
                        "children": [],
                        "name": "Background music",
                        "id": "category_music",
                    },
                    "BACKGROUND_NOISE": {
                        "children": [],
                        "name": "Background noise",
                        "id": "category_noise",
                    },
                    "MULTIPLE_SPEAKERS": {
                        "children": [],
                        "name": "Multiple speakers",
                        "id": "category_multi",
                    },
                },
                "input": "checkbox",
            },
            "instruction": "Audio characteristics",
            "required": 0,
            "isChild": False,
            "level": "asset",
        },
    }
}

In the project settings, Kili labels each job with the level it applies to, so you can check at a glance that your interface is what you intended:

Audio labeling jobs

Creating the project

Audio projects use the AUDIO input type.

project = kili.create_project(
    title="[Kili SDK Notebook]: Audio transcription",
    description="Speaker-attributed transcription of customer support calls",
    input_type="AUDIO",
    json_interface=json_interface,
)

project_id = project["id"]
print("Project ID:", project_id)

Project ID: cmsws1pir04wkvm0w0nh8fqc9

2. Importing audio assets

Audio assets are imported like any other asset type, with append_many_to_dataset.

Kili accepts .mp3, .wav, .flac and .mp4 files.

From a URL

AUDIO_URL = "https://storage.googleapis.com/label-public-staging/demo-projects/audio/EN_Support.mp3"

kili.append_many_to_dataset(
    project_id=project_id,
    content_array=[AUDIO_URL],
    external_id_array=["support_call_en"],
)

From a local file

To upload a file that sits on your machine, pass its path instead of a URL. Note that hosted files and local files cannot be mixed in a single call — use one call for each.

import urllib.request

urllib.request.urlretrieve(AUDIO_URL, "support_call.mp3")

kili.append_many_to_dataset(
    project_id=project_id,
    content_array=["./support_call.mp3"],
    external_id_array=["support_call_en_local"],
)

3. Understanding the audio label format

An audio jsonResponse has one key that no other input type has: speakers.

{
  "speakers": [
    {"id": "spk_agent", "name": "Agent", "color": "#7C3AED"},
    {"id": "spk_customer", "name": "Customer", "color": "#059669"}
  ],
  "TRANSCRIPTION_JOB": {
    "annotations": [
      {
        "mid": "segment_000",
        "startTime": 2.0,
        "endTime": 2.3,
        "speakerId": "spk_customer",
        "text": "Hello."
      }
    ]
  },
  "LANGUAGE_JOB": {"categories": [{"name": "ENGLISH"}]},
  "AUDIO_CHARACTERISTICS_JOB": {"categories": [{"name": "MULTIPLE_SPEAKERS"}]}
}

Speakers

speakers is the cast of the recording. Speakers are defined per label, not per project: two assets in the same project can have completely different speakers, which is exactly what you want when each recording is a different conversation.

Field	Type	Description
`id`	`str`	The identifier you choose. Segments reference the speaker through it.
`name`	`str`	The name displayed on the speaker tag, e.g. `Agent`.
`color`	`str`	Hex color of the tag and of the segment on the waveform, e.g. `#7C3AED`.

All three fields are required. Colors are free-form hex strings; the palette the Kili interface uses when a labeler adds a speaker by hand is #7C3AED, #2563EB, #059669, #DC2626, #D97706, #0891B2, #EC4899, #4F46E5, and picking from it keeps imported labels visually consistent with manually created ones.

Segments

The transcription job holds an annotations list — one entry per segment on the waveform.

Field	Type	Required	Description
`mid`	`str`	yes	Identifier of the segment, unique within the label. It is preserved on export, which makes it the reliable key to join a segment back to your own data.
`startTime`	`float`	yes	Start of the segment, in seconds.
`endTime`	`float`	yes	End of the segment, in seconds.
`text`	`str`	yes	The transcription. Use `""` for a segment that still has to be transcribed.
`speakerId`	`str`	no	Id of one of the entries of `speakers`. Omit it (or set it to `null`) to leave the segment unassigned — it will show up as Unknown in the interface.

Times are expressed in seconds as floats, with millisecond precision — Kili stores them internally as integer milliseconds, so 2.3456 is rounded to 2.346.

Asset-level jobs

Asset-level jobs use the classic jsonResponse shape you already know, keyed by job name: {"categories": [{"name": "ENGLISH"}]} for a classification, {"text": "..."} for a transcription.

4. Importing model pre-annotations

Speech-to-text models paired with a diarization model produce exactly the information Kili needs: time-aligned segments, their transcription, and which speaker uttered them. Importing them as predictions gives labelers a draft to correct instead of a blank waveform.

Here we hardcode the output of such a pipeline, but in a real workflow this would come from Whisper, pyannote.audio, a cloud speech API, or your own model.

speakers = [
    {"id": "spk_agent", "name": "Agent", "color": "#7C3AED"},
    {"id": "spk_customer", "name": "Customer", "color": "#059669"},
]

# (start, end, speaker, text) as produced by a transcription + diarization pipeline
raw_segments = [
    (2.00, 2.30, "spk_customer", "Hello."),
    (3.90, 5.40, "spk_agent", "Hello, I'm speaking to Mariam."),
    (6.40, 7.50, "spk_customer", "Yes, speaking."),
    (8.20, 9.40, "spk_agent", "Hello, my name is Stephen."),
    (9.50, 13.30, "spk_agent", "I'm calling you from the finance department."),
    (
        14.20,
        18.10,
        "spk_agent",
        "You were speaking with Michael before, and your manager is Mr. Omar, correct?",
    ),
    (19.10, 20.10, "spk_customer", "Okay."),
    (
        20.10,
        31.40,
        "spk_agent",
        "All right. I was calling you because we were trying to find a way to make a quick and easy withdrawal of your money back to your bank.",
    ),
    (
        31.70,
        36.30,
        "spk_agent",
        "I think we finally found an option, and that's why I'm calling you.",
    ),
    (36.80, 39.70, "spk_agent", "It will just take another five or ten minutes."),
    (
        40.00,
        44.50,
        "spk_agent",
        "If you're available, I would like to guide you through the steps.",
    ),
    (48.30, 50.40, "spk_agent", "So are you available for me to help you with that?"),
    (51.90, 52.30, "spk_customer", "Yes."),
    (53.00, 54.10, "spk_agent", "Okay, wonderful."),
]

json_response = {
    "speakers": speakers,
    "TRANSCRIPTION_JOB": {
        "annotations": [
            {
                "mid": f"segment_{index:03d}",
                "startTime": start,
                "endTime": end,
                "speakerId": speaker_id,
                "text": text,
            }
            for index, (start, end, speaker_id, text) in enumerate(raw_segments)
        ]
    },
    # asset-level jobs, filled in the same call
    "LANGUAGE_JOB": {"categories": [{"name": "ENGLISH"}]},
    "AUDIO_CHARACTERISTICS_JOB": {"categories": [{"name": "MULTIPLE_SPEAKERS"}]},
}

label_type="PREDICTION" marks the label as model output, and model_name records which model produced it, so you can later compare several models on the same assets.

kili.append_labels(
    project_id=project_id,
    asset_external_id_array=["support_call_en"],
    json_response_array=[json_response],
    label_type="PREDICTION",
    model_name="whisper-large-v3",
)

Kili audio labeling interface

A few things to keep in mind when building the jsonResponse:

mid is mandatory. Unlike bounding boxes in image projects, audio segments are not assigned an identifier automatically; a segment without a mid is rejected.
Segments do not have to be sorted. Kili orders them by startTime when it returns them.
Overlapping segments are allowed, which matters when two people talk over each other.
Every speakerId should exist in speakers. A segment pointing at an unknown speaker is imported, but the interface will render it as Unknown.

To import ground-truth labels rather than predictions, use the very same jsonResponse with the default label_type="DEFAULT".

5. Exporting audio labels

kili.labels returns the labels of a project. Beyond jsonResponse, audio labels expose the speakers relation, which gives you the cast of each label.

labels = kili.labels(
    project_id=project_id,
    asset_external_id_in=["support_call_en"],
    fields=[
        "labelType",
        "modelName",
        "jsonResponse",
        "speakers.id",
        "speakers.name",
        "speakers.color",
    ],
)

label = labels[0]
print(label["labelType"], "-", label["modelName"])
print(json.dumps(label["speakers"], indent=2))

PREDICTION - whisper-large-v3
[
  {
    "id": "spk_agent",
    "name": "Agent",
    "color": "#7C3AED"
  },
  {
    "id": "spk_customer",
    "name": "Customer",
    "color": "#059669"
  }
]

segments = label["jsonResponse"]["TRANSCRIPTION_JOB"]["annotations"]

print(f"{len(segments)} segments")
print(json.dumps(segments[:2], indent=2))

14 segments
[
  {
    "mid": "segment_000",
    "startTime": 2,
    "endTime": 2.3,
    "speakerId": "cmsws1q1j04x8vm0w85204mu8",
    "text": "Hello."
  },
  {
    "mid": "segment_001",
    "startTime": 3.9,
    "endTime": 5.4,
    "speakerId": "cmsws1q1j04x7vm0w5pnofrun",
    "text": "Hello, I'm speaking to Mariam."
  }
]

mid, startTime, endTime and text come back exactly as they were imported — mid in particular is your stable join key back to your own data.

Exporting the whole project to a file

To get every asset and every label at once, use export_labels. The raw and kili formats keep the audio jsonResponse untouched, speakers relation aside; the computer-vision formats (coco, yolo_*, pascal_voc) and geojson do not apply to audio.

By default only submitted labels are exported, so pass label_type_in and export_type="normal" if you also want the predictions.

kili.export_labels(
    project_id=project_id,
    filename="audio_export.zip",
    fmt="raw",
    with_assets=False,
    label_type_in=["DEFAULT", "PREDICTION"],
    export_type="normal",
)

6. Cleanup

Let's remove the project we created for this tutorial.

kili.delete_project(project_id)

Summary

We created an audio project, learned the difference between the segment-level transcription job and asset-level jobs, imported audio assets from a URL and from disk, imported speaker-attributed predictions, and exported them back.

For more on the concepts used along the way, see: