Finding cellular regions with superpixel analysis

Overview:

Whole-slide images often contain artifacts like marker or acellular regions that need to be avoided during analysis. In this example we show how HistomicsTK can be used to develop saliency detection algorithms that segment the slide at low magnification to generate a map to guide higher magnification analyses. Here we show how superpixel analysis can be used to locate hypercellular regions that correspond to tumor-rich content.

This uses Simple Linear Iterative Clustering (SLIC) to get superpixels at a low slide magnification to detect cellular regions. The first step of this pipeline detects tissue regions (i.e. individual tissue pieces) using the get_tissue_mask method of the histomicstk.saliency module. Then, each tissue piece is processed separately for accuracy and disk space efficiency. It is important to keep in mind that this does NOT rely on a tile iterator, but loads the entire tissue region (but NOT the whole slide) in memory and passes it on to skimage.segmentation.slic method. Not using a tile iterator helps keep the superpixel sizes large enough to correspond to tissue boundaries.

Once superpixels are segmented, the image is deconvolved and features are extracted from the hematoxylin channel. Features include intensity and possibly also texture features. Then, a mixed component Gaussian mixture model is fit to the features, and median intensity is used to rank superpixel clusters by ‘cellularity’ (since we are working with the hematoxylin channel).

Note that the decison to fit a gaussian mixture model instead of using K-means clustering is a design choice. If you’d like to experiment, feel free to try other methods of classifying superpixels into clusters using other approaches.

Additional functionality includes contour extraction to get the final segmentation boundaries of cellular regions and to visualize them in HistomicsUI using one’s preferred colormap.

Here are some sample results:

From left to right: Slide thumbnail, superpixel classifications, contiguous cellular/acellular regions

cdetection

Where to look?

|_ histomicstk/
  |_saliency/
     |_cellularity_detection.py
     |_tests/
        |_test_saliency.py
[1]:
import tempfile
import girder_client
import numpy as np
from histomicstk.annotations_and_masks.annotation_and_mask_utils import (
    delete_annotations_in_slide)
from histomicstk.saliency.cellularity_detection_superpixels import (
    Cellularity_detector_superpixels)

import matplotlib.pylab as plt
from matplotlib.colors import ListedColormap
%matplotlib inline

# color map
vals = np.random.rand(256,3)
vals[0, ...] = [0.9, 0.9, 0.9]
cMap = ListedColormap(1 - vals)

Prepwork

[2]:
APIURL = 'http://candygram.neurology.emory.edu:8080/api/v1/'
SAMPLE_SLIDE_ID = '5d586d76bd4404c6b1f286ae'
# SAMPLE_SLIDE_ID = "5d8c296cbd4404c6b1fa5572"

gc = girder_client.GirderClient(apiUrl=APIURL)
gc.authenticate(apiKey='kri19nTIGOkWH01TbzRqfohaaDWb6kPecRqGmemb')

# This is where the run logs will be saved
logging_savepath = tempfile.mkdtemp()

# color normalization values from TCGA-A2-A3XS-DX1
cnorm_thumbnail = {
    'mu': np.array([9.24496373, -0.00966569,  0.01757247]),
    'sigma': np.array([0.35686209, 0.02566772, 0.02500282]),
}
# from the ROI in Amgad et al, 2019
cnorm_main = {
    'mu': np.array([8.74108109, -0.12440419,  0.0444982]),
    'sigma': np.array([0.6135447, 0.10989545, 0.0286032]),
}
[3]:
# deleting existing annotations in target slide (if any)
delete_annotations_in_slide(gc, SAMPLE_SLIDE_ID)

Initialize the cellularity detector

[4]:
print(Cellularity_detector_superpixels.__init__.__doc__)
Init Cellularity_Detector_Superpixels object.

        Arguments:
        -----------
        gc : object
            girder client object
        slide_id : str
            girder ID of slide
        verbose : int
            0 - Do not print to screen
            1 - Print only key messages
            2 - Print everything to screen
            3 - print everything including from inner functions
        monitorPrefix : str
            text to prepend to printed statements
        logging_savepath : str or None
            where to save run logs
        suppress_warnings : bool
            whether to suppress warnings
        cnorm_params : dict
            Reinhard color normalization parameters. Accepted keys: thumbnail
            and main (since thumbnail normalization is different from color
            normalization of tissue at target magnification. Each entry is a
            dict containing values for mu and sigma. This is either given
            here or can be set using self.set_color_normalization_values().
            May be left unset if you do not want to normalize.
        get_tissue_mask_kwargs : dict
            kwargs for the get_tissue_mask() method.
        MAG : float
            magnification at which to detect cellularity
        spixel_size_baseMag : int
            approximate superpixel size at base (scan) magnification
        compactness : float
            compactness parameter for the SLIC method. Higher values result
            in more regular superpixels while smaller values are more likely
            to respect tissue boundaries.
        deconvolve : bool
            Whether to deconvolve and use hematoxylin channel for feature
            extraction. Must be True to ranks spixel clusters by cellularity.
        use_grayscale : bool
            If True, grayscale image is used with SLIC. May be more robust to
            color variations from slide to slide and more efficient.
        use_intensity : bool
            Whether to extract intensity features from the hematoxylin channel.
            This must be True to rank spuerpixel clusters by cellularity.
        use_texture : bool
            Whether to extract Haralick texture features from Htx channel. May
            not necessarily improve results when used in conjunction with
            intensity features.
        keep_feats : list
            Name of intensity features to use. See
            histomicstk.features.compute_intensity_features.
            Using fewer informative features may result in better
            gaussian mixture modeling results.
        n_gaussian_components : int
            no of gaussian mixture model components
        max_cellularity : int
            Range [0, 100] or None. If None, normalize visualization RGB values
            for each tissue piece separately, else normalize by given number.
        opacity : float
            opacity of superpixel polygons when posted to DSA.
            0 (no opacity) is more efficient to render.
        opacity_contig : float
            opacity of contiguous region polygons when posted to DSA.
            0 (no opacity) is more efficient to render.
        lineWidth : float
            width of line when displaying superpixel boundaries.
        cMap : object
            matplotlib color map to use when visualizing cellularity
        visualize_tissue_boundary : bool
            whether to visualize result from tissue detection component
        visualize_spixels : bool
            whether to visualize superpixels, color-coded by cellularity
        visualize_contiguous : bool
            whether to visualize contiguous cellular regions


In this example, and as the default behavior, we use a handful of informative intensity features extracted from the hematoxylin channel after color deconvolution to fit a gaussian mixture model. Empirically (on a few test slides), this seems to give better results than using the full suite of intensity and texture features available. Feel free to experiment with this and find the optimum combination of features for your application.

[5]:
# init cellularity detector
cds = Cellularity_detector_superpixels(
    gc, slide_id=SAMPLE_SLIDE_ID,
    MAG=3.0, compactness=0.1, spixel_size_baseMag=256 * 256,
    max_cellularity=40,
    visualize_spixels=True, visualize_contiguous=True,
    get_tissue_mask_kwargs={
        'deconvolve_first': False,
        'n_thresholding_steps': 2,
        'sigma': 1.5,
        'min_size': 500 },
    verbose=2, monitorPrefix='test',
    logging_savepath=logging_savepath)
Saving logs to: /tmp/tmpt7dygwhf/2019-09-29_18-04.log

Set the color normalization values

You can choose to reinhard color normalize the slide thumbnail and/or the tissue image at target magnificaion. You can either provide the mu and sigma values directly or provide the path to an image from which to infer these values. Please refer to the color_normalization module for reinhard normalization implementation details. In this example, we use a “high-sensitivity, low-specificity” strategy to detect tissue, followed by the more specific cellularity detection module. In other words, the tissue_detection module is used to detect all tissue, and only exclude whitespace and marker. Here we do NOT perform color normalization before tissue detection (empirically gives worse results), but we do normalize when detecting the cellular regions within the tissue.

[6]:
# set color normalization for thumbnail
# cds.set_color_normalization_values(
#     mu=cnorm_thumbnail['mu'],
#     sigma=cnorm_thumbnail['sigma'], what='thumbnail')

# set color normalization values for main tissue
cds.set_color_normalization_values(
    mu=cnorm_main['mu'], sigma=cnorm_main['sigma'], what='main')

Run the detector

[7]:
print(cds.run.__doc__)
Run cellularity detection and optionally visualize result.

        This runs the cellularity detection +/- visualization pipeline and
        returns a list of CD_single_tissue_piece objects. Each object has
        the following attributes

        tissue_mask : np array
            mask of where tissue is at target magnification
        ymin : int
            min y coordinate at base (scan) magnification
        xmin : int
            min x coordinate at base (scan) magnification
        ymax : int
            max y coordinate at base (scan) magnification
        xmax : int
            max x coordinate at base (scan) magnification
        spixel_mask : np array
            np array where each unique value represents one superpixel
        fdata : pandas DataFrame
            features extracted for each superpixel. Index corresponds to
            values in the spixel_mask. This includes a 'cluster' column
            indicatign which cluster this superpixel belongs to.
        cluster_props : dict
            properties of each superpixel cluster, including its assigned
            cellularity score.


[8]:
tissue_pieces = cds.run()
test: set_slide_info_and_get_tissue_mask()
test: Tissue piece 1 of 2
test: Tissue piece 1 of 2: set_tissue_rgb()
test: Tissue piece 1 of 2: set_superpixel_mask()
test: Tissue piece 1 of 2: set_superpixel_features()
test: Tissue piece 1 of 2: set_superpixel_assignment()
test: Tissue piece 1 of 2: assign_cellularity_scores()
test: Tissue piece 1 of 2: visualize_individual_superpixels()
test: Tissue piece 1 of 2: Posting doc 1 of 5
test: Tissue piece 1 of 2: Posting doc 2 of 5
test: Tissue piece 1 of 2: Posting doc 3 of 5
test: Tissue piece 1 of 2: Posting doc 4 of 5
test: Tissue piece 1 of 2: Posting doc 5 of 5
test: Tissue piece 1 of 2: visualize_contiguous_superpixels()
test: Tissue piece 1 of 2: Posting doc 1 of 5
test: Tissue piece 1 of 2: Posting doc 2 of 5
test: Tissue piece 1 of 2: Posting doc 3 of 5
test: Tissue piece 1 of 2: Posting doc 4 of 5
test: Tissue piece 1 of 2: Posting doc 5 of 5
test: Tissue piece 2 of 2
test: Tissue piece 2 of 2: set_tissue_rgb()
test: Tissue piece 2 of 2: set_superpixel_mask()
test: Tissue piece 2 of 2: set_superpixel_features()
test: Tissue piece 2 of 2: set_superpixel_assignment()
test: Tissue piece 2 of 2: assign_cellularity_scores()
test: Tissue piece 2 of 2: visualize_individual_superpixels()
test: Tissue piece 2 of 2: Posting doc 1 of 5
test: Tissue piece 2 of 2: Posting doc 2 of 5
test: Tissue piece 2 of 2: Posting doc 3 of 5
test: Tissue piece 2 of 2: Posting doc 4 of 5
test: Tissue piece 2 of 2: Posting doc 5 of 5
test: Tissue piece 2 of 2: visualize_contiguous_superpixels()
test: Tissue piece 2 of 2: Posting doc 1 of 5
test: Tissue piece 2 of 2: Posting doc 2 of 5
test: Tissue piece 2 of 2: Posting doc 3 of 5
test: Tissue piece 2 of 2: Posting doc 4 of 5
test: Tissue piece 2 of 2: Posting doc 5 of 5

Check the results

The resultant list of objects correspond to the results for each “tissue piece” detected in the slide. You may explore various attributes like the offset coordinates, tissue mask, superpixel labeled mask, superpixel feature data, and superpixel cluster properties.

[9]:
plt.imshow(tissue_pieces[0].tissue_mask, cmap=cMap)
[9]:
<matplotlib.image.AxesImage at 0x7f7d1a3c0c50>
../_images/examples_semantic_segmentation_superpixel_approach_15_1.png
[10]:
plt.imshow(tissue_pieces[0].spixel_mask, cmap=cMap)
[10]:
<matplotlib.image.AxesImage at 0x7f7d1c35ad10>
../_images/examples_semantic_segmentation_superpixel_approach_16_1.png
[11]:
tissue_pieces[0].fdata.head()
[11]:
Intensity.Mean Intensity.Median Intensity.Std Intensity.IQR Intensity.HistEntropy cluster
5 86.639486 74.0 50.329524 34.0 1.775608 4
6 73.010711 68.0 26.269239 20.0 1.341687 4
67 88.820514 73.0 56.899073 49.0 1.942993 5
68 72.959455 67.0 31.681785 21.0 1.349226 4
71 100.068075 79.0 60.721122 67.0 1.989196 5
[12]:
tissue_pieces[0].cluster_props
[12]:
{1: {'cellularity': 20, 'color': 'rgb(253,253,255)'},
 2: {'cellularity': 37, 'color': 'rgb(167,0,0)'},
 3: {'cellularity': 47, 'color': 'rgb(127,0,0)'},
 4: {'cellularity': 26, 'color': 'rgb(255,105,105)'},
 5: {'cellularity': 29, 'color': 'rgb(255,29,29)'}}

Check the visualization on HistomicsUI

Now you may go to the slide on Digital Slide Archive and check the posted annotations.