Quantitative measurement and assessment of magnetic resonance images (MRI) has an important role in the diagnosis, evaluation, and follow-up of brain tumors. Unfortunately, defining tumor boundaries on such images is a difficult task for automated methods – and even trained radiologists – as tumors tend to have irregular shapes, ill-defined margins, and various degrees of heterogeneity. In this work, we demonstrate that binary pixel classification of tumor regions is sub-optimal in reflecting the underlying biology of different tumor compartments and propose that pixel-wise probability maps are a more appropriate representation of tumor extent, providing volumetric approximations with error estimates. We present a framework that performs multimodal analysis of a tumor boundary using different perspectives to identify regions of high heterogeneity and that, together with error analysis, can be used as reference to identify instances on which the tumor segmentation is unclear and can be employed to improve performance of the segmentation algorithm on future cases. This approach was evaluated on a set of 110 cases of high grade gliomas obtained from The Cancer Genome Atlas (TCGA) for which a manual tumor region of interest (ROI) was available. From these cases, our algorithm automatically generated binary and probability masks for the dataset; comparison with the manual ROI resulted in a Dice coefficient of 0.8. It was qualitatively observed that the derived probability maps (overlaid as color maps on an MRI volume) provide a better visualization of tumor burden and its heterogeneous characteristics, which can be valuable during treatment planning (e.g., radiotherapy) and evaluation of disease progression over time.