DeepLabCut

PyPI version PyPI - Downloads Krihelimeter GitHub stars GitHub forks

Generic badge License: LGPL v3 Image.sc forum Gitter Twitter Follow

DeepLabCut is a toolbox for markerless pose estimation of animals performing various tasks. In 2018, we demonstrated the capabilities for trail tracking, reaching in mice and various Drosophila behaviors during egg-laying (see Mathis et al. for details). There is, however, nothing specific that makes the toolbox only applicable to these tasks and/or species. The toolbox has already been successfully applied (by us and others) to rats, humans, various fish species, bacteria, leeches, various robots, cheetahs, mouse whiskers and race horses. DeepLabCut utilizes the feature detectors (ResNets + readout layers) of one of the state-of-the-art algorithms for human pose estimation by Insafutdinov et al., called DeeperCut, which inspired the name for our toolbox (see references below). Furthermore, we have added faster variants with MobileNetV2 backbones (see Pretraining boosts out-of-domain robustness for pose estimation). Additionally, we have improved the inference speed and provided additional augmentation methods (via tensorpack and imgaug).

www.deeplabcut.org

VERSION 2.x+: This is the Python package of DeepLabCut that was originally released with our Nature Protocols paper (preprint here). This package includes graphical user interfaces to label your data, and take you from data set creation to automatic behavioral analysis. It also introduces an active learning framework to efficiently use DeepLabCut on large experimental projects, and data augmentation tools that improve network performance, especially in challenging cases (see panel b).

VERSION 1.0: The initial, Nature Neuroscience version of DeepLabCut can be found in the history of git, or here: https://github.com/AlexEMG/DeepLabCut/releases/tag/1.11

Installation

How to install DeeplabCut

Documentation: The DeepLabCut Process

An overview of the pipeline and workflow for project management.

For a step-by-step user guide, please read the Nature Protocols paper!

DEMO the code

We provide several Jupyter Notebooks: one that walks you through a demo dataset to test your installation, and another Notebook to run DeepLabCut from the beginning on your own data. We also show you how to use the code in Docker, and on Google Colab.

Why use DeepLabCut?

Top left: Due to transfer learning it requires little training data for multiple, challenging behaviors (see Mathis et al. 2019 for details). Top Right: Video analysis is fast (see Mathis et al. 2019 and Mathis/Warren for details). Mid Left: The feature detectors are robust to video compression (see Mathis/Warren for details). Mid Right: It allows 3D pose estimation with a single network and camera (see Mathis/Warren). Bottom: It allows 3D pose estimation with a single network trained on data from multiple cameras together with standard triangulation methods (see Nath* and Mathis* et al. for details).

Code contributors:

DLC code was originally developed by Alexander Mathis & Mackenzie Mathis, and extended in 2.0 with Tanmay Nath. The feature detector code is based on Eldar Insafutdinov’s TensorFlow implementation of DeeperCut. DeepLabCut is an open-source tool and has benefited from suggestions and edits by many individuals including Mert Yuksekgonul, Tom Biasi, Richard Warren, Ronny Eichler, Hao Wu, Federico Claudi, Gary Kane and Jonny Saunders as well as the contributors. Please see AUTHORS for more details!

This is an actively developed package and we welcome community development and involvement.

Community Support, Developers, & Help:

References:

If you use this code or data please cite Mathis et al, 2018 and, if you use, the Python package (DeepLabCut2.x) please also cite Nath, Mathis et al, 2019. If you utilize the MobileNets please cite Mathis et al. 2019.

Please check out the following references for more details:

@article{Mathisetal2018,
    title={DeepLabCut: markerless pose estimation of user-defined body parts with deep learning},
    author = {Alexander Mathis and Pranav Mamidanna and Kevin M. Cury and Taiga Abe  and Venkatesh N. Murthy and Mackenzie W. Mathis and Matthias Bethge},
    journal={Nature Neuroscience},
    year={2018},
    url={https://www.nature.com/articles/s41593-018-0209-y}
}

 @article{NathMathisetal2019,
    title={Using DeepLabCut for 3D markerless pose estimation across species and behaviors},
    author = {Nath*, Tanmay and Mathis*, Alexander and Chen, An Chi and Patel, Amir and Bethge, Matthias and Mathis, Mackenzie W},
    journal={Nature Protocols},
    year={2019},
    url={https://doi.org/10.1038/s41596-019-0176-0}
}

@article{insafutdinov2016eccv,
    title = {DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model},
    author = {Eldar Insafutdinov and Leonid Pishchulin and Bjoern Andres and Mykhaylo Andriluka and Bernt Schiele},
    booktitle = {ECCV'16},
    url = {http://arxiv.org/abs/1605.03170}
}

Our open-access pre-prints:

@misc{Mathis2019_pretraining,
    title={Pretraining boosts out-of-domain robustness for pose estimation},
    author={Alexander Mathis and Mert Y\"uksekg\"on\"ul and Byron Rogers and Matthias Bethge and Mackenzie W. Mathis},
    year={2019},
    eprint={1909.11229},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
    }

@article{NathMathis2018,
    author = {Nath*, Tanmay and Mathis*, Alexander and Chen, An Chi and Patel, Amir and Bethge, Matthias and Mathis, Mackenzie W},
    title = {Using DeepLabCut for 3D markerless pose estimation across species and behaviors},
    year = {2018},
    doi = {10.1101/476531},
    publisher = {Cold Spring Harbor Laboratory},
    URL = {https://www.biorxiv.org/content/early/2018/11/24/476531},
    eprint = {https://www.biorxiv.org/content/early/2018/11/24/476531.full.pdf},
    journal = {bioRxiv}
    }

@article{mathis2018markerless,
    title={Markerless tracking of user-defined features with deep learning},
    author={Mathis, Alexander and Mamidanna, Pranav and Abe, Taiga and Cury, Kevin M and Murthy, Venkatesh N and Mathis, Mackenzie W and Bethge, Matthias},
    journal={arXiv preprint arXiv:1804.03142},
    year={2018}
    }

@article{MathisWarren2018speed,
    author = {Mathis, Alexander and Warren, Richard A.},
    title = {On the inference speed and video-compression robustness of DeepLabCut},
    year = {2018},
    doi = {10.1101/457242},
    publisher = {Cold Spring Harbor Laboratory},
    URL = {https://www.biorxiv.org/content/early/2018/10/30/457242},
    eprint = {https://www.biorxiv.org/content/early/2018/10/30/457242.full.pdf},
    journal = {bioRxiv}
    }

License:

This project is licensed under the GNU Lesser General Public License v3.0. Note that the software is provided “as is”, without warranty of any kind, express or implied. If you use the code or data, please cite us!.

News (and in the news):