UDMT tutorial

Installation

Environment

  • Ubuntu 20.04 + (required)
  • Python 3.8
  • Pytorch 1.7.1
  • NVIDIA GPU (GeForce RTX 3090) + CUDA (11.7)

Environment Configuration

  1. Create a virtual environment and install PyTorch.

    $ conda create -n udmt python=3.8
$ conda activate udmt
$ pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
$ sudo apt-get install ninja-build
$ sudo apt-get install libturbojpeg

  2. We made a installable pip release of UDMT [pypi]. You can install it by entering the following command:

    $ pip install udmt-pip

2. For Windows

Environment

  • Windows 10
  • Python 3.8
  • Pytorch 1.7.1
  • NVIDIA GPU (GeForce RTX 3090) + CUDA (11.0)

Environment Configuration

  1. Create a virtual environment and install PyTorch. In the 3rd step, please select the correct Pytorch version that matches your CUDA version from https://pytorch.org/get-started/previous-versions/.

    $ conda create -n udmt python=3.8
$ conda activate udmt
$ pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

  2. We made a installable pip release of UDMT [pypi]. You can install it by entering the following command:

    $ pip install udmt-pip

  3. Install Precise ROI pooling: If your environment is the same as ours, directly copy <UDMT_install_path>\udmt\env_file\prroi_pool.pyd to <Anaconda_install_path>\anaconda3\envs\udmt\Lib\site-packages. Otherwise, build prroi_pool.pyd file with Visual Studio with the tutorial.

  4. Install libjpeg-turbo: You can download installer from the official libjpeg-turbo Sourceforge repository, install it and copy <libjpeg-turbo_install_path>\libjpeg-turbo64\bin\turbojpeg.dll to the directory from the system PATH C:\Windows\System32.

GUI tutorial

1. Start UDMT in the terminal

  1. Once you have UDMT installed, start by opening a terminal. Activate the environment and download the codes with:

    $ source activate udmt
$ git clone https://github.com/cabooster/UDMT
$ cd UDMT/

  2. To launch the GUI, simply enter in the terminal:

    $ python -m udmt.gui.launch_script

  3. Pre-trained models will be downloaded automatically before launching the GUI. Alternatively, you can manually download the model and place it in the specified location.

    Model name Location
    trdimp_net_ep.pth.tar ./udmt/gui/pretrained
    XMem.pth ./udmt/gui/tabs/xmem/saves
    sam_vit_b_01ec64.pth ./udmt/gui/tabs/xmem/sam_model

2. Create a new project

  1. Click the Create New Project button.

  2. Set the Project Path and Project Name.

  3. Add videos by clicking the Browse Videos button, then select the folders containing your videos. By default, the selected videos will be copied to the project path.

  4. Finally, click the Create button to complete this process. The project you created will open automatically.

Open an existing project

  1. To reopen a project, click the Load Project button on the home page.

  2. Navigate to your project folder and select the config.yaml file.

Directory structure

Click to unfold the directory tree 

UDMT
|-- config.yaml
|-- videos
    |-- video1.mp4
    |-- video2.mp4
|-- training-datasets
    |-- video1
        |-- img
        	|-- ...
        |-- label
        	|-- ...
|-- models
    |-- video1
        |-- DiMPnet_ep0018.pth.tar
        |-- DiMPnet_ep0019.pth.tar
        |-- DiMPnet_ep0020.pth.tar
|-- tracking-results
    |-- video1
        |-- video1-whole-filter5.mp4
        |-- video1-whole-filter5.npy
        |-- tmp-videos
        	|-- ...
|-- tmp
    |-- video1
        |-- evaluation_metric.json
        |-- evaluation_metric_for_test.json
        |-- evaluation_metric_for_train.json
        |-- extracted-images
            |-- start_pos_array.txt
            |-- 00000.jpg
			|-- ...
        |-- images
        	|-- ...
        |-- masks
        	|-- ...
        |-- test_set_results
        	|-- ...
        |-- train_set_results
        	|-- ...

Folder description

project_name/videos: Path to save the videos to be processed.

project_name/training-datasets: Path to save the training datasets.

project_name/models: Path to save the models.

project_name/tracking-results: Path to save the tracking results, including .npy files and .MP4 files showing the tracking trajectories.

project_name/tmp: Path to save temporary files, including images used during tracking initialization, extracted frames, and files generated during automatic parameter tuning.

3. Tracking initialization

  1. Click the Select Videos button to choose a video you want to process (you can select videos from the videos folder in your project file).

  2. After selecting the video, click the Launch Tracking Initialization GUI button to open the sub-GUI.

  3. In the sub-GUI, click on the centroids of all the animals you want to track.

  4. Click Forward Propagate to start the foreground extraction process.

  5. Wait until foreground extraction is completed for all frames. After foreground extraction, check whether all animals are marked with a red mask. If many consecutive frames fail to extract an animal (which usually happens with small animals like flies), please increase ‘memory frame every’ to 10,000. Once finished, close the sub-GUI to proceed to the next tab.

4. Create training dataset

  1. Click the Select Videos button to choose a video you want to process (if you have already selected videos in the previous tab, you don’t need to select again).

  2. After selecting the video, choose the parameters for preprocessing the videos.

  3. Click the Create Training Dataset button to start the creation process.

  4. During the process, you can click the Show Video button at any time to visualize the creation progress.

  5. Once the creation is complete, a popup message will appear, notifying you to proceed to the next tab.

5. Train network

  1. Click the Select Videos button to choose a video you want to process (if you have already selected videos in the previous tab, you don’t need to select them again).

  2. Set the training parameters:

    Batch Size: Select a larger batch size if you have multiple GPUs. If you encounter a “CUDA out of memory” error, reduce the batch size.

    Number of Workers: Set to 0 on Windows; on Linux, you can set it to 8.

    Max Training Epochs: The default is 20 epochs. This value doesn’t need to be changed unless necessary.

  3. Click the Train Network button to start the training process. Note: Due to thread occupation issues during network training, the training phase output logs will be displayed in the console instead of the GUI log window.

6. Analyze videos

  1. Select a video you want to process.

  2. Choose the parameters for preprocessing the videos and set the Filter Size for post-processing smooth trajectory filtering.

  3. Click the Analyze Videos button to start the tracking process.

  4. During the tracking, you can click the Show Video button at any time to visualize the tracking process. (This includes automatic parameter tuning and the final tracking process.)

  5. Once the tracking is complete, the result files will be automatically saved in the tracking-results folder.

  6. A popup message will appear, notifying you to proceed to the next tab.

7. Visualize tracks

  1. Select a video you want to process.

  2. Choose the parameters for post-processing the videos.

  3. Click the Launch Track Visualization GUI button to start the visualization.

  4. In the pop-up sub-GUI: The left side shows the current frame and the right side displays the trajectory XY coordinates that change with each frame.

  5. You can click the Frame bar to visualize each frame.

  6. Adjust the Dot Size bar to change the size of the dots marking positions in the image.

GUI demo video