An optimized image analysis algorithm for detecting nuclear signals in digital whole slides for histopathology

Róbert Paulik, Tamás Micsik, Gábor Kiszler, Péter Kaszál, János Székely, Norbert Paulik, Eszter Várhalmi, Viktória Prémusz, T. Krenács, B. Molnár

Research output: Contribution to journalArticle

12 Citations (Scopus)


Nuclear estrogen receptor (ER), progesterone receptor (PR) and Ki-67 protein positive tumor cell fractions are semiquantitatively assessed in breast cancer for prognostic and predictive purposes. These biomarkers are usually revealed using immunoperoxidase methods resulting in diverse signal intensity and frequent inhomogeneity in tumor cell nuclei, which are routinely scored and interpreted by a pathologist during conventional light-microscopic examination. In the last decade digital pathology-based whole slide scanning and image analysis algorithms have shown tremendous development to support pathologists in this diagnostic process, which can directly influence patient selection for targeted- and chemotherapy. We have developed an image analysis algorithm optimized for whole slide quantification of nuclear immunostaining signals of ER, PR, and Ki-67 proteins in breast cancers. In this study, we tested the consistency and reliability of this system both in a series of brightfield and DAPI stained fluorescent samples. Our method allows the separation of overlapping cells and signals, reliable detection of vesicular nuclei and background compensation, especially in FISH stained slides. Detection accuracy and the processing speeds were validated in routinely immunostained breast cancer samples of varying reaction intensities and image qualities. Our technique supported automated nuclear signal detection with excellent efficacy: Precision Rate/Positive Predictive Value was 90.23 ± 4.29%, while Recall Rate/Sensitivity was 88.23 ± 4.84%. These factors and average counting speed of our algorithm were compared with two other open source applications (QuPath and CellProfiler) and resulted in 6–7% higher Recall Rate, while 4- to 30-fold higher processing speed. In conclusion, our image analysis algorithm can reliably detect and count nuclear signals in digital whole slides or any selected large areas i.e. hot spots, thus can support pathologists in assessing clinically important nuclear biomarkers with less intra- and interlaboratory bias inherent of empirical scoring.

Original languageEnglish
Pages (from-to)595-608
Number of pages14
JournalCytometry Part A
Issue number6
Publication statusPublished - Jun 1 2017


  • cell nucleus detection algorithm
  • DAPI stain
  • ER
  • fluorescence
  • histopathology
  • immunohistochemistry
  • Ki-67
  • PR
  • whole slide analysis

ASJC Scopus subject areas

  • Pathology and Forensic Medicine
  • Histology
  • Cell Biology

Fingerprint Dive into the research topics of 'An optimized image analysis algorithm for detecting nuclear signals in digital whole slides for histopathology'. Together they form a unique fingerprint.

  • Cite this

    Paulik, R., Micsik, T., Kiszler, G., Kaszál, P., Székely, J., Paulik, N., Várhalmi, E., Prémusz, V., Krenács, T., & Molnár, B. (2017). An optimized image analysis algorithm for detecting nuclear signals in digital whole slides for histopathology. Cytometry Part A, 91(6), 595-608.