Volumetric rendering of multimodality, multivariable medical imaging data

KEYWOFID: Volume rendering, magnetic resonance imaging, positron emission tomography, magnetic resonance angiography, image processing, computer graphics. (e.g. local measurements of metabolism). Therefore, it is desirable to combine these data in one single 3-D display so that one can visualize and utilize these types of information simultaneously. We have developed a method for simultaneous display of brain surface anatomy from MR data and surface metabolic activity from PET data [Levin 89b,89c]. Thus, the highly resolved MR images were used to create a 3-D atlas of each patient's brain anatomy for the purpose of localizing poorly-resolved PET measurements of brain metabolism. In the next section, the details of the technique are presented along with a clinical example, illustrating its application to planning surgery from treatment of medically intractable epilepsy. INTFIODtlCTION There are well-known techniques for using X-ray computed tomography (CT) data to create 3-D views of the human skeleton. In earlier work [Levin 89a], we described a new technique for using magnetic resonance (MR) images to create 3-D views of the brain surface. The 3-D images clearly showed surface abnormalities as well as important anatomical landmarks which could not be identified on cross-sectional views (see Figure 2a). Although this type of image shows brain surface anatomy well, it does not provide functional information as well as other anatomical information such as vascular morphology. In this paper, we report our work on integrated 3-D display of data from multiple crosssectional imaging modalities, including MR, CT, and PET as well as our work on the visualization of vascular structure. The data from different imaging modalities are usually complementary. For example, MR images are the best for delineating soft tissue anatomy, and CT images are optimal for depicting bones and calcifications. On the other hand, PET images provide functional information MR imaging is an inherently multivariable technique since it is sensitive to multiple intrinsic tissue parameters which can be measured by using different pulse sequences. For example, "angiographic" techniques can be used to produce images in which blood vessels are highlighted. Currently, maximum intensity projection (MIP) is widely used to produce projection views from 3-D data sets of this type. MIP is basically a ray tracing technique, in which the maximum intensity along each ray is retained in the projection regardless of where the maximum occurs [Rossnick 86]. Despite its simplicity, the technique works reasonably well. However, the MIP method is sensitive to susceptibility artifacts and chemical shift artifacts because these artifacts usually have intensity above that of the background. Furthermore it does not present a realistic 3D model of vascular structure since it is not based on a physically meaningful computer graphics model and stationary tissues are not well depicted by this technique. Some authors have attempted to render vascular and stationary anatomy by means of sophisticated surface reconstruction techniques such as dividing cubes and marching cubes [Cline 88, Cline 89]. The resulting display was marked by lack of detail, probably due to the inherent limitations of surface rendering. In addition, the vessels inside the brain could not be seen simultaneously with brain surface anatomy. Recently, Hohne et al. [Hohne 89] have introduced a method by viewing a portion of vascular structure through a cut on the brain surface. With this method, one may be confused by not having a complete view of both structures. In order to avoid these problems, we have developed a novel technique, based on volume rendering, for displaying vessels in isolation or in conjunction with brain surface CH Volume Visualization Workshop 45 anatomy. The third section is devoted to the discussion and illustration of this technique. INTEGRATED DISPLAY OF MR AND PET The procedure for producing the combined display consists of five steps: (1) image editing or processing, (2) spatial registration of the MR and PET images, (3) volume rendition of brain surface anatomy from edited MR data, (4) calculation of the average surface metabolic activity from PET images, (5) use of colors to map the metabolic measurements onto the surface of the 3-D model of the brain. The flow chart in Figure 1 illustrates the general scheme of the technique. acquired MR images ] [original PET images J [ V~lr~n~ur~ cdeered j integrated 3-D display I of MR and PET j Fig. 1. Schematic diagram of the algorithm to produce the integrated MR and PET display. 46 CH Volume Visualization Workshop Each cross-sectional MR image is subjected to an interactive editing program of our own design, which serves to isolate the brain and surrounding cerebrospinal fluid (CSF). This software uses a threshold tracking algorithm to draw a contour between the surface of the brain and the inner surface of the skull in each slice image. The operator specifies a "seed" point to initiate the contour generation for each image, and he may have to perform some manual editing when the contour algorithm fails. Notice that this contour need not follow the complex convolutions of the brain's surface, since the surface of the brain will be delineated automatically by a volume-rendering process in which CSF is made invisible (transparent); thus, we need not know the exact boundary of the brain and our contour can be very flexible. Therefore this editing step is much easier to accomplish than conventional surface extraction. Normally, MR and PET studies of a patient are conducted at different times and patient positioning and other geometric parameters of the two studies may differ. It is therefore necessary to register each PET image voxel to the corresponding MR image voxel. This is achieved with a retrospective technique developed by Pelizzari and Chen [Pelizzari 87, Levin 88a]. The basic idea is to fit a crude surface model derived from one set of data to the corresponding surface extracted from the other set of data by means of a geometric transformation which includes rotation, translation and scaling. In this application, surface models of the brain were used. The transformation is then used to resample the PET data at the exact positions of the MR slices. The major advantages of this technique are that (1) no fiducial marks or other prospective maneuvers are needed and (2) the fitting is not sensitive to errors in the individual surface points since an entire surface is used. Thus a coarse surface model (e.g. tile or wire frame) is sufficient. Volume rendition of the brain's surface is accomplished by applying a 3-D volume rendering [Drebin 88] program based on ChapVolumes routines on a Pixar Image Computer (Pixar, San Rafael, California). A continuous lookup table is applied to the grey scale value of every voxel in order to estimate the fractions of brain and CSF within it. Each voxel is then assigned color and opacity values, which depend on these fractions and the attributes of the "pure" brain (white, opaque) and CSF (colorless, transparent). Similarly, each voxel is assigned a density value for calculating gradients used to shade the model. 3D views are generated by shading the volume with simulated light, by applying depth-encoding, and by tracing rays through the volume. For general descriptions of volume rendering, the reader is referred to [Drebin 88, Levoy 88]. In most cases, 64 views at equally spaced angles are produced for movie loop presentation. Since the image registration is only accurate within several millimeters and the definition of the exact contour of the brain is subject to errors, we cannot accurately measure surface metabolic activity by the PET intensity at the location of a single MR voxel on the surface of the brain. Rather, we use the PET intensity averaged over 510 mm of brain near the surface. Notice that this does not really degrade our PET data because its resolution is usually around 1 cm. In order to calculate the average surface PET activity, we first derive a rim-like mask within the cortex from the edited MR image. This is realized by (1) thresholding the edited image to produce the region of "brain" in each slice, (2) "eroding" the brain by blurring and thesholding, and (3) taking the difference between the original and "eroded" images. The width of the resulting rim can be controlled by the amount of blurring. Multiplying this mask by the corresponding registered PET slice images results in masked PET slices, which are subsequently used as input to a ray tracing program to produce 3-D projections of the average PET intensity in 64 angular views. Each intensity value in the 3-D model of average surface PET activity is then assigned a color. An integrated display of MR and PET is then created by assigning each point on the brain surface model a color corresponding to the PET color model. Portions of the brain not scanned by PET remain uncolored. These techniques are illustrated by considering the case of a patient with intractable seizures. The original MR images were obtained from a single 10-minute volumetric pulse sequence on our 1.5 Tesla unit (Magnetom, Siemens Medical Systems, Inc., Iselin, New Jersy). This exam produced images of 63 contiguous slices with 3 mm thickness; each image was a 256 x 256 matrix of square pixels with 1.2 mm sides. No abnormality was visible on these or other cross-sectional MR images. Figure 2a shows a 3-D view of the brain without coloring by PET data. It clearly shows important anatomical structures such as the motor strip, sensory strip and speech area. In addition, the gyri of the lower motor and sensory strips are abnormally flattened; this finding could not be appreciated on the cross-sectional views. Figure 2b is the 3-D color model of the average cortex PET activity. The pink region shows an abnormal area with hypermetabolic activity where the seizure activity