Analyzing remotely hosted images with the girder client

This example describes how to run image analysis tasks in a workflow process on collections of slides hosted on a DSA server. The API is used to retrieve pixel data from the server and the analysis is performed locally, with results pushed back to the server as anotations for visualization. Note that this is not the same approach as using girder tasks to run jobs remotely as a user through HistomicsUI. This is a utility intended for use by developers of image analysis and machine learning algorithms.

In this example, we will be running a cellularity detection workflow on all slides in the following source girder directory and the results are posted to the following results girder directory.

Where to look?

|_ histomicstk/
   |_workflows/
      |_workflow_runner.py
      |_specific_workflows.py
      |_tests/
         |_test_workflow_runner.py
[1]:
import shutil
import tempfile
import girder_client
# import numpy as np
from pandas import read_csv
from histomicstk.workflows.workflow_runner import Slide_iterator
# from histomicstk.saliency.cellularity_detection import (
#     Cellularity_detector_superpixels)
from histomicstk.saliency.cellularity_detection_thresholding import (
    Cellularity_detector_thresholding)
from histomicstk.workflows.workflow_runner import (
    Workflow_runner, Slide_iterator)
from histomicstk.workflows.specific_workflows import (
    cellularity_detection_workflow)

Connect girder client and set analysis parameters

[2]:
APIURL = 'http://candygram.neurology.emory.edu:8080/api/v1/'
SAMPLE_SOURCE_FOLDER_ID = "5d5c28c6bd4404c6b1f3d598"
SAMPLE_DESTINATION_FOLDER_ID = "5d9246f6bd4404c6b1faaa89"

# girder client
gc = girder_client.GirderClient(apiUrl=APIURL)
# gc.authenticate(interactive=True)
gc.authenticate(apiKey='kri19nTIGOkWH01TbzRqfohaaDWb6kPecRqGmemb')

# This is where the run logs will be saved
logging_savepath = tempfile.mkdtemp()

# params for cellularity thresholding
cdt_params = {
    'gc': gc,
    'slide_id': '',  # this will be handled by the slide iterator
    'GTcodes': read_csv('../../histomicstk/saliency/tests/saliency_GTcodes.csv'),
    'MAG': 3.0,
    'visualize': True,
    'verbose': 2,
    'logging_savepath': logging_savepath,
}

Explore the docs

Workflow_runner

You will need to use the Workflow_runner() object.

[3]:
print(Workflow_runner.__init__.__doc__)
Init Workflow_runner object.

        Arguments
        -----------
        slide_iterator : object
            Slide_iterator object
        workflow : method
            method whose parameters include slide_id and monitorPrefix,
            which is called for each slide
        workflow_kwargs : dict
            keyword arguments for the workflow method
        kwargs : key-value pairs
            The following are already assigned defaults by Base_HTK_Class
            but can be passed here to override defaults
            [verbose, monitorPrefix, logging_savepath, suppress_warnings]


Note how this requires: - Slide_iterator instance - which yields information about the slides you want to the run the workflow on. - workflow - a method that you define, which runs on a single slide. - workflow_kwargs - parameters for your defined method.

In this example, we will be using cellularity_detection_workflow() as our workflow to run, which is defined in the histomicstk.workflows.specific_workflows module.

Slide_iterator

[4]:
print(Slide_iterator.__init__.__doc__)
Init Slide_iterator object.

        Arguments
        -----------
        gc : object
            girder client object
        source_folder_id : str
            girder ID of folder in which slides are located
        keep_slides : list
            List of slide names to keep. If None, all are kept.
        discard_slides : list
            List of slide names to discard.
        kwargs : key-value pairs
            The following are already assigned defaults by Base_HTK_Class
            but can be passed here to override defaults
            [verbose, monitorPrefix, logger, logging_savepath,
            suppress_warnings]


Specific workflow to run

[5]:
print(cellularity_detection_workflow.__doc__)
Run cellularity detection for single slide.

    The cellularity detection algorithm can either be
    Cellularity_detector_superpixels or Cellularity_detector_thresholding.

    Arguments
    -----------
    gc : object
        girder client object
    cdo : object
        Cellularity_detector object instance. Can either be
        Cellularity_detector_superpixels() or
        Cellularity_detector_thresholding(). The thresholding-based workflow
        seems to be more robust, despite being simpler.
    slide_id : str
        girder id of slide on which workflow is done
    monitoPrefix : str
        this will set the cds monitorPrefix attribute
    destination_folder_id : str or None
        if not None, copy slide to this girder folder and post results
        there instead of original slide.
    keep_existing_annotations : bool
        keep existing annotations in slide when posting results?


Initialize the workflow runner

[6]:
# Init specific workflow (Cellularity_detector_thresholding)
cdt = Cellularity_detector_thresholding(**cdt_params)

# Init workflow runner
workflow_runner = Workflow_runner(
    slide_iterator=Slide_iterator(
        gc, source_folder_id=SAMPLE_SOURCE_FOLDER_ID,
        # keep_slides=None),  # run all slides in girder directory
        keep_slides=[  # run specific slides only
            'TCGA-A1-A0SK-01Z-00-DX1_POST.svs',
            'TCGA-A2-A04Q-01Z-00-DX1_POST.svs',
        ]),
    workflow=cellularity_detection_workflow,
    workflow_kwargs={
        'gc': gc,
        'cdo': cdt,
        'destination_folder_id': SAMPLE_DESTINATION_FOLDER_ID,
        'keep_existing_annotations': False, },
    logging_savepath=cdt.logging_savepath,
    monitorPrefix='test')
Saving logs to: /tmp/tmpz7t9bo3m/2019-10-27_18-53.log
Saving logs to: /tmp/tmpz7t9bo3m/2019-10-27_18-53.log

Run the detector

[7]:
workflow_runner.run()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): copying slide to destination folder
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): set_slide_info_and_get_tissue_mask()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: set_tissue_rgb()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: initialize_labeled_mask()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: assign_components_by_thresholding()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: -- get HSI and LAB images ...
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: -- thresholding blue_sharpie ...
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: -- thresholding blood ...
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: -- thresholding whitespace ...
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: color_normalize_unspecified_components()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: -- macenko normalization ...
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: find_potentially_cellular_regions()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: find_top_cellular_regions()
test: slide 1 of 2 (TCGA-A1-A0SK-01Z-00-DX1_POST.svs): Tissue piece 1 of 1: visualize_results()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): copying slide to destination folder
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): set_slide_info_and_get_tissue_mask()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: set_tissue_rgb()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: initialize_labeled_mask()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: assign_components_by_thresholding()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: -- get HSI and LAB images ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: -- thresholding blue_sharpie ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: -- thresholding blood ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: -- thresholding whitespace ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: color_normalize_unspecified_components()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: -- macenko normalization ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: find_potentially_cellular_regions()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: find_top_cellular_regions()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 1 of 2: visualize_results()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: set_tissue_rgb()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: initialize_labeled_mask()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: assign_components_by_thresholding()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: -- get HSI and LAB images ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: -- thresholding blue_sharpie ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: -- thresholding blood ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: -- thresholding whitespace ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: color_normalize_unspecified_components()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: -- macenko normalization ...
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: find_potentially_cellular_regions()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: find_top_cellular_regions()
test: slide 2 of 2 (TCGA-A2-A04Q-01Z-00-DX1_POST.svs): Tissue piece 2 of 2: visualize_results()

Check the DSA/HistomicsTK visualization

Now you may go to the Digital Slide Archive and check the posted results at the results girder directory.