Action segmentation

Action segmentation#

Installation:#

First, install DLC2Action based on the installation guide.

We note, that you can also use it in COLAB (see below), which requires no installation on your system.

Using DLC2Action on an example labeled dataset#

We prepared Jupyter Notebooks for going through an example dataset from Sturman et al. [SvZSchlappi+20].

You can find the Notebook here Note you can also open and run this notebook with Colab.

Advanced features#

DLC2Action has many additional features, you can explore those by:

checking out a more detailed notebook (also based on the Sturman et al. data). Note you can also open and run this notebook with .
or checking out the documentation

Running DLC2Action on your data#

You will need to have some annotation data. DLC2action accepts many different formats. If you have not yet annotated any behaviors, then you can use the following GUI.

Note

For good performance you will need to annotate multiple examples for each behavior that you are interested in. This will usually take a bit of time!

Basic Workflow (Python commands)#

Import DLC2Action

from dlc2action.project import Project

(1) Create a project 📂

PROJECT_NAME = "test"  # the name of the project
PROJECTS_PATH = ...  # the path to the projects folder
DATA_TYPE = ...  # choose from Project.print_data_types()
ANNOTATION_TYPE = ...  # choose from Project.print_annotation_types()
DATA_PATH = ...  # path to data files
ANNOTATION_PATH = ...  # path to annotation files

FILE_PATHS = (
    None  # set to a list of file paths if you want to generate predictions for new data
)

project = Project(
    PROJECT_NAME,
    projects_path=PROJECTS_PATH,
    data_type=DATA_TYPE,
    annotation_type=ANNOTATION_TYPE,
    data_path=DATA_PATH,
    annotation_path=ANNOTATION_PATH,
)

# run project.help() for general information
# and project.help("data") to make sure that your files are organized correctly

**(2) Update parameters (model, learning rate, etc.) **

project.update_parameters(
    ...
)  # run project.list_blanks() to see which parameters you need to update

MODELS = [
    "c2f_tcn",
    "transformer",
]  # see project.help("models") for a list of available models

(3) Perform hyperparameter search and train your model

for model in MODELS:  # run a hyperparameter search for your models
    project.run_default_hyperparameter_search(
        f"{model}_search",
        model_name=model,
        metric="f1",
    )

for model in MODELS:  # train models with the best hyperparameters
    project.run_episode(
        f"{model}_best",
        load_search=f"{model}_search",
        parameters_update={"general": {"model_name": model}},
        n_seeds=3,
        force=True,
    )

project.plot_episodes(  # compare training curves
    [f"{model}_best" for model in MODELS],
    metrics=["f1"],
    title="Best model training curves",
)

(4) Evaluate your model

for model in MODELS:  # evaluate more metrics
    project.evaluate(
        [f"{model}_best"],
        parameters_update={
            "general": {"metric_functions": ["segmental_f1", "pr-auc", "f1"]},
            "metrics": {"f1": {"average": "none"}},
        },
    )

project.get_results_table(
    [f"{model}_best" for model in MODELS]
)  # get a table of the results

BEST_MODELS = [MODELS[0], MODELS[1]]  # choose the best model

project.run_prediction(
    [
        f"{model}_best" for model in BEST_MODELS
    ],  # if you choose multiple models, the predictions will be averaged
    force=True,
    file_paths=FILE_PATHS,  # make a prediction for new data
)

project.list_episodes(  # get the experiment history
    display_parameters=[
        "results/f1",
        "meta/time",
        "meta/training_time",
        "general/model_name",
    ],
    value_filter=f"results/f1::>0.3,general/model_name::{BEST_MODELS[0]}",
)

References

[SvZSchlappi+20]

Oliver Sturman, Lukas von Ziegler, Christa Schläppi, Furkan Akyol, Mattia Privitera, Daria Slominski, Christina Grimm, Laetitia Thieren, Valerio Zerbi, Benjamin Grewe, and others. Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology, 45(11):1942–1952, 2020.

Let’s go back 🔙.