Paper: Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov. “Unsupervised learning of video representations using lstms.” International conference on machine learning. 2015.

• Basics
  • Gradient vanishing problem.
    • Solution: LSTM (Long short term Memory)
  • Unsupervised learning model
    • Crucial to have the right inductive biases and right objective functions.
• Main methods
  • LSTM encoder to get representations of videos
  • Multi-LSTM decoders for different tasks.  — Produce a target sequence
    • Different choices of target sequences.
      • Same as input
      • To predict the future
  • Inputs: — two kinds
    • Image patches
    • High level percepts
  • Simple squared loss function as a starting point
  • Encoder-decoder RNN that can be used with any loss functions.
  • LSTM details
    • Cell: memory unit
    • Input date, forget date and output date
  • LSTM autoencoder model
    • Two RNNs: encoder and decoder
    • Input: a sequence of vectors
    • After the last input been read, the decoder start to do prediction and output the sequence
    • Output: Same as input sequence, reverse order
  • LSTM future predictor model
    • Predict only the next frame at each time step
  • Combination of the above two models
• Take home messages
• Other methods mentioned — for video representations learning
  • Supervised learning: 3D convolutional nets.
  • Unsupervised learning:
    • ICA (Independence subspace analysis)
    • Generative models for understanding transformations between 2 consecutive images
    • Generative model for predicting next frame or interpolate between frames — Important to choose right loss function — Not square loss function