vp_suite.models.phydnet

class PhyDNet(device, **model_kwargs)

Bases: vp_suite.base.base_model.VPModel

This class implements the PhyDNet prediction model, as introduced by Le Guen and Thome in https://arxiv.org/abs/2003.01460 and implemented in https://github.com/vincent-leguen/PhyDNet. PhyDNet aims to disentangle physical dynamics such as movement parameters from so-called ‘residual’ dynamics such as appearance. For the physical dynamics, the PhyCell performs PDE-Constrained prediction in latent space. For the residual dynamics, a modified version of the ConvLSTM cell is used that permits recurrent steps one frame at a time.

CAN_HANDLE_ACTIONS = True

Whether the model can handle actions or not.

CODE_REFERENCE = 'https://github.com/vincent-leguen/PhyDNet'

The code location of the reference implementation.

MATCHES_REFERENCE: str = 'Not Yet'

A comment indicating whether the implementation in this package matches the reference.

NAME = 'PhyDNet'

The model’s name.

PAPER_REFERENCE = 'https://arxiv.org/abs/2003.01460'

The publication where this model was introduced first.

__init__(device, **model_kwargs)

Initializes the model by first setting all model hyperparameters, attributes and the like. Then, the model-specific init will actually create the model from the given hyperparameters

Parameters

  • device (str) – The device identifier for the module.

  • **model_kwargs (Any) – Model arguments such as hyperparameters, input shapes etc.

convlstm_hidden_dims = [128, 128, 64]

Channel dimensionality per ConvCell layer

convlstm_kernel_size = (3, 3)

ConvCell kernel size

convlstm_n_layers = 3

Number of ConvCell layers

encoder_fwd(frame, action, first_timestep=False, decoding=False)
forward(x, pred_frames=1, **kwargs)

Given an input sequence of t frames, predicts pred_frames (p) frames into the future.

Parameters

  • x (torch.Tensor) – A batch of b sequences of t input frames as a tensor of shape [b, t, c, h, w].

  • pred_frames (int) – The number of frames to predict into the future.

  • () (**kwargs) –

Returns: A batch of sequences of p predicted frames as a tensor of shape [b, p, c, h, w].

moment_loss_scale = 1.0

Scaling factor for the moment loss (for PDE-Constrained prediction by the PhyCells)

phycell_channels = 49

Channel dimensionality for the PhyCells

phycell_kernel_size = (7, 7)

PhyCell kernel size

phycell_n_layers = 1

Number of PhyCell layers

pred_1(x, **kwargs)

Given an input sequence of t frames, predicts one single frame into the future.

Parameters

  • x (torch.Tensor) – A batch of b sequences of t input frames as a tensor of shape [b, t, c, h, w].

  • **kwargs (Any) – Optional input parameters such as actions.

Returns: A single frame as a tensor of shape [b, c, h, w].

teacher_forcing_decay = 0.003

Per-Episode decrease of the teacher forcing ratio (Starts out at 1.0)

train_iter(config, data_loader, optimizer, loss_provider, epoch)

PhyDNet’s training iteration utilizes a scheduled teacher forcing ratio. Otherwise, the iteration logic is the same as in the default train_iter() function.

Parameters

  • config (dict) – The configuration dict of the current training run (combines model, dataset and run config)

  • data_loader (DataLoader) – Training data is sampled from this loader.

  • optimizer (Optimizer) – The optimizer to use for weight update calculations.

  • loss_provider (PredictionLossProvider) – An instance of the LossProvider class for flexible loss calculation.

  • epoch (int) – The current epoch.