Next-Gen Dataset: Exclude background patches during training

[jdeschamps]

Problem

In microscopy, images often contain large background areas. During training, this will cause the network to spend substantial amount of time learning "nothing". Being able to exclude background regions would be a useful feature that many users would probably enjoy.

Potential solutions

Solution 1: CSBDeep-style

There is no reason not to copy what has been developed before in CSBDeep:

# parameters:
# threshold
# percentile
from scipy.ndimage import maximum_filter
def _filter(datas, patch_size, dtype=np.float32):
    image = datas[0]
    if dtype is not None:
        image = image.astype(dtype)
    # make max filter patch_size smaller to avoid only few non-bg pixel close to image border
    patch_size = [(p//2 if p>1 else p) for p in patch_size]
    filtered = maximum_filter(image, patch_size, mode='constant')
    return filtered > threshold * np.percentile(image,percentile)
return _filter

Source

Solution 2: use mean and std

Exclude patches whose mean is N standard deviation away from the dataset mean.

Relevant other issues

#292