stillleben

YCB-Video example

What can it do for me?

Stillleben generates realistic arrangements of rigid bodies and provides various outputs that can be used to train deep learning models:

  • A rendered RGB image of the scene. Depending on the rendering settings, this rendering can be of quite high quality, in some cases photorealistic. The image can be imbued with noise using the stillleben.camera_model module.
  • A depth image.
  • Class-wise and instance-wise semantic segmentation of the scene.
  • Camera- and object-space coordinates, highly useful for training methods that exploit pixel-wise correspondence.
  • Camera-space normals.

Stillleben is highly integrated with the PyTorch deep learning framework. It can be used to produce training data on-line to save dataset generation time or to train methods that require interaction with the scene.

For scene registration and other applications we provide an approximative differentiation module in stillleben.diff which allows backpropagation of image-space gradients to object poses and shapes.

This page documents the Python API of stillleben. The Python API is a wrapper around the C++ core implementation.

Getting started

If you haven't already, go through the installation instructions.

Here is a short API example:

import stillleben as sl
import torch
from PIL import Image

sl.init() # use sl.init_cuda() for CUDA interop

# Load a mesh
mesh = sl.Mesh(SL_PATH / 'tests' / 'stanford_bunny' / 'scene.gltf')

# Meshes can come in strange dimensions - rescale to something reasonable
mesh.scale_to_bbox_diagonal(0.5)

# Create a scene with a few bunnies
scene = sl.Scene((1920,1080))

for i in range(10):
    obj = sl.Object(mesh)
    scene.add_object(obj)

# Let them fall in a heap
scene.simulate_tabletop_scene()

# Setup lighting
scene.choose_random_light_position()
scene.ambient_light = torch.tensor([0.3, 0.3, 0.3])

# Display a plane & set background color
scene.background_plane_size = torch.tensor([3.0, 3.0])
scene.background_color = torch.tensor([0.1, 0.1, 0.1, 1.0])

# Render a frame
renderer = sl.RenderPass()
result = renderer.render(scene)
print('Resulting RGB frame:', result.rgb().shape)
print('Resulting segmentation frame:', result.instance_index().shape)

# Save as JPEG
Image.fromarray(result.rgb()[:,:,:3].cpu().numpy()).save('rgb.jpeg')

# Display interactive viewer
sl.view(scene)