Paper:

Newell, Alejandro, Zhiao Huang, and Jia Deng. “Associative embedding: End-to-end learning for joint detection and grouping.” Advances in Neural Information Processing Systems. 2017.

• Performance
• Basics
  • Associative embedding
  • Jointly perform detections and grouping using a single-stage deep network trained end-to-end
  • For each detection, introduce a “tag” (is a number) to identify which group this detection belongs to.
    • Note: We have no ground truth tags for the network to predict, because what matters is not the particular tag values, only the difference between them.
  • Output: Two heatmaps
    • A heatmap for Per-pixel detection scores. (detection score at each pixel for each joint.)
    • A heatmap for per-pixel identity tags.(tagging score at each pixel for each joint.)
    • For multi-person pose estimation, output a detection heatmap and a tagging heatmap for each body joint, then group body joints with similar tags into individual people.
  • Two loss functions together
    • Detection loss: mean square error (MSE) between each predicted detection heatmap and its ground truth heatmap (is a 2D Gaussian activation at each keypoint location).
    • Grouping loss: We compare the tags within each person and across people, Tags within a person should be the same, while tags across people should be different.

• Other methods mentioned.
  • Vector embedding
  • Perceptual organization: group pixels of an image into regions, parts and objects.
  • Multiplerson pose estimation
  • Instance segmentation
• Evaluation
  • Dataset
    • MPII human multi-person http://human-pose.mpi-inf.mpg.de/
      • 25K images containing over 40K people with annotated body joints. Covers 410 human activities.
    • COCO 2016 keypoints challenge. http://cocodataset.org/#keypoints-challenge2017
  • Evaluation metrics
    • Average precision (AP)
• Questions: How to get the tags of training data?

Paper:

Zhao, Ruiqi, Yan Wang, and Aleix M. Martinez. “A simple, fast and highly-accurate algorithm to recover 3d shape from 2d landmarks on a single image.” IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).

Paper:

Zhou, Xingyi, et al. “Deep kinematic pose regression.” European Conference on Computer Vision. Springer, Cham, 2016.

Paper:

Tekin, Bugra, et al. “Structured prediction of 3d human pose with deep neural networks.” arXiv preprint arXiv:1605.05180(2016).

Paper:

Chen, Xianjie, and Alan Yuille. “Parsing occluded people by flexible compositions.” Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015.

Paper:

Sun, Xiao, et al. “Compositional human pose regression.” The IEEE International Conference on Computer Vision (ICCV). Vol. 2. 2017.

Key: Structure-aware

Paper:

Liang, Guoqiang, et al. “A limb-based graphical model for human pose estimation.” IEEE Transactions on Systems, Man, and Cybernetics: Systems (2017).

• • • Code not available
    • Caffe
    • NVIDIA Tesla K40m GPU
    • Basics
      • New task: Human limb detection
        • Detect and represent the local image appearance.
      • Use human limbs to augment constraints between neighboring human joints.
      • Design a new limb representation: Model a limb as a wide line.
    • Main method: ConvNet consists of two modules: Limbs and joints detector, and a limb-based graphical model. Both output heatmaps and trained with Euclidean distance loss.
      • Unified framework detector: VGG16 architecture.
        • Human limb detection combined with joint localization
        • Integrate the two detection processes in a single CNN
      • After initial detections, a two-steps graphical model.
        • To capture the spatial relationship among human joints. And to capture the spatial relationship among limb in a coarse to fine way.
        • First step: Full-connected graphical model is used to capture the coarse relation from an arbitrary
        • Second step: Construct a new pairwise relation term based on limbs.
    • Other methods mentioned
      • Define the relationship as geometric constraint on the relative locations of two neighboring joints.
        • Not using the local appearance (image input itself) of the region connecting two neighboring joints
        • Lead to problems: double-counting and localization failure.
      • PS model (Pictorial Structures)
        • Most popular and influential model.
        • Model human limb as a rigid oriented rectangle
        • Model human limb as bar, detect it by searching parallel edges.
        • Model a limb with 2 joints. Or add an extra joint at the middle point.
        • Use image segmentation methods to distinguish limbs from background.
      • ConvNet based pose estimation
        • Extract appearance and type score.
        • Heat-map
          • Heat-map based methods are per-pixel classification problems with large contextual information.
        • Use Conv-Net to learn a MRF-based graphical model.
      • Add motion feature
      • For Spatial relations:
        • Tree structure.
      • Appearance and relation models.
        • The relation among human parts is defined as geometric constraints  on the location and orientation of parts.
          • Spring like model
          • Conditional probability of joints location
        • Note: For joints with higher flexibility， the constraint is too weak.
      • Graphical model over parts.
        • Nodes representing parts
        • Edges encoding constraints.
        • Note: limited by hand-crafted features and tree-based graphical models, the accuracy was not good.

• • Limb modeling:
• • Evaluation
    • PCP 74.6 on LSP
    • Dataset: FLIC, LSP

Paper:

Newell, Alejandro, Zhiao Huang, and Jia Deng. “Associative embedding: End-to-end learning for joint detection and grouping.” Advances in Neural Information Processing Systems. 2017.

Paper:

Papandreou, George, et al. “Towards accurate multiperson pose estimation in the wild.” arXiv preprint arXiv:1701.01779 8 (2017).

key:

ResNet for keypoint, heatmap and offset

• Performance
• Basics
  • Without ground truth of the location or the scale of the person.
  • Top-down approach
• Main methods
  • Pipeline:
    • Person box detection using Faster-RCNN, (ResNet-101)
      • CNN backbone pre-trained on ImageNet
        • No multi-scale evaluation.
    • Person pose estimation. Use ResNet 101 for heatmap and offset.
      • K=17 keypoints.
      • Classification && Regression
        • Firstly classify whether it is (0 or 1) in the neighborhood of any keypoint. (heatmap)
        • Predict a 2D local offset vector to see the precise keypoint location.

• Take home message
• Other methods mentioned
• Evaluation
  • Object Keypoint Similarity
  • COCO