MultiNet with Transformers: A Model for Cancer Diagnosis Using ImagesJournal: arXiv

Abstract

Cancer is a leading cause of death in many countries. An early diagnosis of cancer based on biomedical imaging ensures effective treatment and a better prognosis. However, biomedical imaging presents challenges to both clinical institutions and researchers. Physiological anomalies are often characterized by slight abnormalities in individual cells or tissues, making them difficult to detect visually. Traditionally, anomalies are diagnosed by radiologists and pathologists with extensive training. This procedure, however, demands the participation of professionals and incurs a substantial cost. The cost makes large-scale biological image classification impractical. In this study, we provide unique deep neural network designs for multiclass classification of medical images, in particular cancer images. We incorporated transformers into a multiclass framework to take advantage of data-gathering capability and perform more accurate classifications. We evaluated models on publicly accessible datasets using various measures to ensure the reliability of the models. Extensive assessment metrics suggest this method can be used for a multitude of classification tasks.

1 Introduction

Breast cancer affects one out of every eight women in their lifetime[Tao et al., 2015]. Preventing breast cancer requires early detection and diagnosis[Shaikh et al., 2021]. There is currently no magic bullet for the automated detection of breast cancer. For diagnostic and therapeutic purposes, artificial intelligence (AI) has been widely used in biomedical imaging[Shin et al., 2016]. To complete a diagnosis, well-trained specialists usually identify subtleties in cell abnormalities [Panayides et al., 2020]. The capability to identify cell abnormalities requires long-term training. Biomedical imaging has not been widely adopted in some medical facilities, particularly those with limited healthcare resources, due to the high cost of training.[Panayides et al., 2020]. As a cost-cutting measure, AI-based diagnosis has been proposed to reduce costs and increase accuracy. AI approaches have proven to be effective diagnostic tools for biomedical images [Yu et al., 2018] . Many studies have achieved expert-level disease detection accuracy [Rajpurkar et al., 2018, Tschandl et al., 2019, Haggenmüller et al., 2021], including breast cancer[Alakhras et al., 2015, Rodriguez-Ruiz et al., 2019], skin cancer[Brinker et al., 2019, Esteva et al., 2017], pneumonia[Rajpurkar et al., 2017, Basu et al., 2020], and hip fracture[Gale et al., 2017], as well as other medical fields.

Deep learning is an artificial intelligence implementation that can convert large amounts of images into deep domain knowledge for image-based diagnosis [Yu et al., 2018]. Deep learning incorporates image data into multiple neural network layers to mimic how medical imaging experts learn from experience. To classify disease based on images, these layers convert patterns to digital signals. A specialized diagnostic model labels biomedical images with disease types. Because of the ability to classify biomedical images, deep learning-based classifiers have been widely used in diagnosing diseases from biomedical images[Panayides et al., 2020, McBee et al., 2018].

Despite deep learning’s effectiveness in simulating human learning processes, a number of issues remain unresolved. First, existing deep learning models are heavily biased toward a specific type of disease, which does not reflect real-world applications. Numerous studies have demonstrated that deep learning can classify numerous types of diseases using binary classification. There is a disconnection between models’ binary categorization and an expert’s real-world activity of identifying all possible concerns for many diseases based on a single image. This issue may be resolved by the use of multiple classifications for images [Miotto et al., 2018, Serag et al., 2019].

A deep learning model can perform multiple classifications by acquiring knowledge from images depicting various diseases. However, due to the difficulty of detecting regions of interest in biomedical images, learning to classify multiple diseases needs a deliberate model design. Convolutional neural networks (CNNs) and their modifications have been developed, including VGG-16, VGG-19, Resnet, and Inception. While these models worked admirably for multiple categorizations of general objects, they are unable to meet the demand for multiple classifications of diagnoses inherent in nature. Due to the tiny variance, high dimensionality, and various modalities present in medical imaging, the accuracy of the model of multiple diagnosis disease classification decreases rapidly [Simonyan and Zisserman, 2014, He et al., 2016, Szegedy et al., 2016]. Current neural network structures are unable to detect small regions of interest from biomedical images. As a result, developing a new neural network structure while keeping unimportant features may be the key to multiple classification of biomedical images.

Previous studies [Che et al., 2021, Wu et al., 2021] have shown that a transformer network can successfully address the following biomedical image classification challenges: First, existing AI approaches are not suitable for classifying a variety of disorders. Second, it is indeed not possible to use the current models’ architecture to repeatedly classify images of different diseases during training. Our suggested approach not only obtains minor image features that other deep learning models often neglect but also processes images of varying quality, structure, and region of interest.

The purpose of this study is to develop a method for identifying and describing the various anomalies associated with breast cancer. Furthermore, our model improved the ability to detect various types of carcinoma, resulting in a more clinically applicable scenario. Our mode can bridge the gap between Transformer’s development and its limited application in medical imaging, specifically in classifying breast cancer sub-types. We used the BreakHis dataset in this study to solve a multiclass classification assignment for eight different types of cancer.

The following contributions are listed:

• Initially, we integrated the descriptive capabilities of global and local information of ViT’s and CNN’s into a single model called MultiNet-ViT. This capability ensures a more distinct feature representation for distinguishing biological image types.

• Second, the model incorporates the concept of multiscale image analysis to characterize the details of histopathology images under a microscope, thereby increasing the model’s generalizability to different magnification factors.

• A comprehensive implementation of numerous models is presented. In addition, the model’s superior performance and generalizability are compared against additional models.

2 Transformer-Based Models

2.1 Background

Transformer[Vaswani et al., 2017], an alternative to convolutional neural networks, has dominated the field of natural language processing (NLP), including speech recognition [Dong et al., 2018], synthesis [Li et al., 2019b], text to speech translation [Vila et al., 2018], and natural language generation [Topal et al., 2021]. As a example of deep learning architectures, Transformer was first introduced to handle sequential inference tasks in NLP. While recurrent neural networks (RNNs) [Graves et al., 2013] (e.g., long short-term memory network (LSTM) [Sak et al., 2014]) explicitly use a sequence of inference processes, Transformers capture long-term dependencies of sequential data with stacked self-attention layers. In this manner, Transformer is efficient because they solve the sequential learning problem in one shot and effective by stacking very deep models. Several Transformer architectures trained on large-scale architectures have become widely popular in solving NLP tasks such as BERT [Devlin et al., 2018] and GPT [Radford et al., 2018, Brown et al., 2020] etc.

Inspired by the success of Transformers in NLP, [Dosovitskiy et al., 2020] proposed the Vision Transformer (ViT) by formulating image classification as a sequence prediction task of the image patch (region) sequence, thereby capturing long-term dependencies within the input image. ViT and its derived instances have achieved state-of-the-art performance on several benchmark datasets. Transformers have become very popular across a wide spectrum of computer vision tasks, including image classification [Dosovitskiy et al., 2020], detection [Carion et al., 2020], segmentation [Zheng et al., 2021], generation [Parmar et al., 2018], and captioning [Li et al., 2019a]. Furthermore, Transformers also play an important role in video-based applications [Zhou et al., 2018]. Since 2017, Transformers have been used for a variety of computer vision tasks, including general image recognition [Touvron et al., 2021, Matsoukas et al., 2021], object detection [Carion et al., 2020, Zhu et al., 2020], segmentation [Ye et al., 2019], image classification [Che et al., 2021], image super-resolution [Yang et al., 2020], video interpretation [Sun et al., 2019, Girdhar et al., 2019], image generation [Chen et al., 2021b], test-to-image integration [Ramesh et al., 2021], and visual question answering [Tan and Bansal, 2019, Su et al., 2019].

Transformers have recently been adopted in the field of medical image analysis for disease diagnosis [Gao et al., 2021, Zhang and Wen, 2021] and other clinical purposes. For instance, the works in [Costa et al., 2021, Tulder et al., 2021] utilized Transformers to distinguish COVID-19 from other types of pneumonia using computed tomography (CT) or X-ray images, meeting the urgent need of detecting COVID-19 patients fast and effectively. Besides, Transformers were successfully applied to image segmentation [Zhang et al., 2021], detection [Xie et al., 2021], and synthesis [Watanabe et al., 2021], remarkably achieving state-of-the-art results. Despite the fact that studies have been devoted to customizing Transformers for medical image analysis tasks, such customization has raised new challenges that have yet to be resolved. Many studies have developed efficient Transformers [Jaszczur et al., 2021, Li et al., 2022, Liu et al., 2021] while maintaining high performances.

Although Transformers performed well in a variety of natural language tasks, their application in biomedical imaging is limited. For example, some preliminary work is shown in the Deformable DETR [Zhu et al., 2020] model, which is used for object detection. Although many studies are not designed for biomedical images, we believe the attention mechanism also applies to the biomedical imaging field. Max-Deeplab [Wang et al., 2021] is the first end-to-end model for panoptic segmentation that demonstrates how Transformers can be used to predict the mask with a category label. A recent co-attention Transformer-based study [Chen et al., 2021c] demonstrates that several whole slide images can be integrated for patient survival prediction to achieve superior results.Inspired by approaches in Visual Question Answering that can attribute how word embeddings attend to salient objects in an image, the attention mechanism can focus on histology patches that are important predictors of patient survival rate. Other biomedical image classification attempts [Chen et al., 2021a, Che et al., 2021, Zou et al., 2021], segmentation [Prangemeier et al., 2020, Yun et al., 2021], and other clinical outcome predictions [Chen et al., 2021c, Zhang et al., 2020, Shickel et al., 2021] have been used. A Transformer network is thought to perform significantly better in multi-model tasks and compensates for the shortcomings of CNN features. The new self-attention mechanism retains long-term features from biomedical image datasets.

2.2 Vision Transformer(ViT)

Recently, Vision Transformers (ViT) have achieved highly competitive performance in benchmarks for several computer vision applications, including image classification, object detection, and semantic image segmentation. Vision Transformer is a model that applies the Transformer to the image classification task and was proposed in 2020 [Dosovitskiy et al., 2020]. The model architecture is nearly identical to the original Transformer, but with a twist to allow images to be treated as input in a similar format of natural language processing.

ViT divides the image into N “patches” of such as 16$\times$16. Since the images are coming in the form of (Height$\times$Width$\times$ number of Channels), $X\in\mathbb{R}^{H\times W\times C}$, they cannot be handled directly by a transformer that deals with (1D) sequences, so it flattens them and makes a linear projection to convert them into 2D patches in the form of $x_{p}\in\mathbb{R}^{N\times(P^{2}.C)}$, Where $N$ is computed using the Equation 1, and $P$ is the size of each image patch. So each patch can be treated as a token, which can be input to the Transformer.

$$\begin{split}N=\frac{H\times W}{P^{2}}\end{split}$$ (1)

In addition, ViT uses the strategy of pre-training first and then fine-tuning. ViT is pre-trained with JFT-300M, a dataset containing 300 million images, and then fine-tuned for downstream tasks such as ImageNet. ViT is the first pure transformer model to achieve SotA performance on ImageNet, and this has led to a massive surge in research on transformers as applied to computer vision tasks.

Training ViT requires a large amount of data. Transformers are less accurate with less data, but show a quicker accuracy increase with more data, and outperform CNNs when pre-trained on the JFT-300M.

ViT may not benefit from this characteristic because medical datasets are often small. Regarding this problem, a recent investigation by Christos Matsoukas [Matsoukas et al., 2021] has compared the advantages of ViT and CNN. They concluded that (1) ViTs pre-trained on ImageNet perform comparably to CNNs when data is limited. (2) Transfer learning favors ViTs when applying standard training protocols and settings. (3) ViTs outperform their CNN counterparts when self-supervised pre-training is followed by supervised fine-tuning. Therefore, these findings suggest that medical image analysis can benefit from the use of ViTs. At the same time, using ViT gained improved explainability because of the attention mechanism, which can highlight the image patches that are important for the classification task. These properties on CNNs are not available. Therefore, we believe ViT is a superior model for biomedical image analysis.

2.3 Data-efficient image transformers (DeiT)

One drawback of the ViT transformer is that it does not generalize well when trained on insufficient amounts of data. To overcome the problem that the ViT model must train on large amounts of data. In a paper named “Training data-efficient image transformers $\&$ istillation through attention”, Hugo Touvron, Matthieu Cord, et al. proposed a convolution-free transformer network, DeiT, that achieves top-1 accuracy of 83.1% on ImageNet with no external data. The training was completed on a single 8-GPU node in less than 3 days. DeiT introduces a new teacher-student strategy specific to transformers that relies on a distillation token, similar to the class token already employed in transformer networks.

Compared to the ViT model which requires hundreds of millions of images, DeiT, however, can be trained easily with approximately 1.2 million images. To attain that goal, they implemented the following strategies:

• The first key ingredient of DeiT is its training strategy. Initially, researchers used data augmentation, optimization, and regularization to simulate training on a much larger data set, as done in CNN.

• Further, they modified the Transformer architecture to allow native distillation. (Distillation is a process by which one neural network (the student NN) learns from the output of another network (the teacher NN)).

• They used a CNN as a teacher model for the Transformer. Using distillation may hamper the performance of neural networks. So, the student model learns from two different sources that may diverge: from a labeled data set (strong supervision) and the teacher.

To alleviate the problem, a distillation token is introduced: a learned vector that flows through the network along with the transformed image data and cues the model for its distillation output, which can differ from the token’s class output. This improved distillation method is specific to transformers.

Because of these distillation methods, DeiT performs significantly better than the ViT model on a relatively small biomedical data set. This increases the possibility of using a relatively small dataset.

3 Methodology

3.1 BreakHis dataset

Table 1 shows the distribution of the BreakHis dataset on multiclass classification. Depending on the microscopic appearance of the tumor cells, both benign and malignant breast cancers can be subdivided into a number of sub-types, each with an unique prognosis and treatment outcome.. The dataset presently comprises four histologically distinct types of benign breast tumors: Adenosis (A), Fibroadenoma (F), Phyllodes Tumor (PT), and Tubular Adenoma (TA); and four histologically unique types of malignant tumors (breast cancer): Ductal carcinoma (DC), Lobular Carcinoma (LC), Mucinous Carcinoma (MC), and Papillary Carcinoma (PC).

Table 1: The Distribution of the BreakHis dataset by Magnification Factors and Categories- Multiclass Classification Setting

Class	Sub-Class	Magnifications Factor				Total
		40X	100X	200X	400X
Benign	F	253	260	264	237	1014
	A	114	113	111	106	444
	TA	149	150	140	130	569
	PT	109	121	108	115	453
Malignant	DC	864	903	896	788	3451
	LC	156	170	163	137	626
	MC	205	222	196	169	792
	PC	145	142	135	138	560
Total		1995	2081	2013	1820	7909

3.2 Model Architecture

The MultiNet model, as shown in Figure 1, entails combining many networks in order to achieve a wide range of aims. Specifically, (1) two transfer learning models, VGG19 and ResNet, are used as the framework’s backbone. (2) Since the networks operate in parallel and features are extracted by different networks at different times rather than entering immediately into the fully linked, the inadequacies of one network are offset by another. Then these networks are merged together like the approach used in C-Net [Barzekar and Yu, 2022]. The architecture of the middle, network, and inner network is similar to the C-Net model.

Immediately, Convolutional (Conv) Layers generated by the inner network are fed into a couple of more Conv layers with the following structure; A filter size of 1$\times$1 and stride of 1, the same padding, and 1024 filters followed by an extra 1024 filter with the size of 3$\times$3 and stride of 1, the same padding and a 2$\times$2 max-pooling. These 1024 Conv layers are then convolved through three more Conv layers with 768 as the number of filters for each, however, the filter size for the first Conv layer is 1$\times$1 and the rest are 3$\times$3.

After the preceding Conv layers, the output is fed into a small multi-layer perceptron (MLP) consisting of two linear layers, each followed by a Dropout layer, and a final linear layer with 1024 units and eight outputs (number of classes). All of the linear layers use ReLU activation functions, except for the last one, which uses a Softmax function as shown in Equation 2.

$$\begin{split}\sigma(\vec{\mathbf{x}})_{i}=\frac{e^{x_{i}}}{\sum_{j=1}^{C}e^{x_{j}}}\end{split}$$ (2)

where $\vec{\mathbf{x}}$ represents the input vector and C the total number of classes

The MLP head of the MultiNet model is then concatenated, $\oplus$, with the MLP head of the ViT model to classify eight different classes on BreakHis dataset.

For the proposed model, we use a cross-entropy function as the loss function, as shown Equation 3 below:

$$\begin{split}L(\hat{y},y)=-\left(\sum_{i=1}^{N}y_{i}\log\hat{y}_{i}+(1-y_{i})\log(1-\hat{y}_{i})\right)\end{split}$$ (3)

where $y_{i}$ denotes the $i^{th}$ label $y$ out of N classes, and $\hat{y}$ represents the $i^{th}$ output from the model.

Figure 1: MultiNet-ViT Architecture: Integrating MultiNet with ViT

4 Result and Discussion

The multiclass classification involving eight classes is conducted. We provide a confusion matrix and following metrics including: precision and recall, f1 score, and classification accuracy to assess the performance of the model. Recall is the percentage of images that were successfully classified from the ground truth, whereas precision represents the percentage of images that were accurately classified into that specific predicted class. The F1 score takes into account both precision and recall. The accuracy is the proportion of successfully predicted images (classified images) relative to the total number of predictions.

The model employs a categorical cross-entropy, Equation 3 loss function. For loss function minimization, the Adam optimizer is used to fine-tune the weight parameters in order to achieve the best results. $1{e}^{-4}$ is selected as the learning rate, and $\beta_{1}$, $\beta_{2}$ is set to 0.9,0.999 respectively. The batches for training, validating, and testing are all set to have a size of 8.

4.1 ViT-based models

Leveraging evaluation metrics like accuracy and F1-Score, we present the success of the architecture of ViT-based models in the succeeding results and discussions.

Let us start with ViT model first. The accuracy on each class is shown in 2(d) employing the ViT model alone on the entire dataset combining all the magnification factor including 40X, 100X, 200X, and 400X. As it shows the model achieve 100% accuracy on adenosis and papillary carcinoma classes.

(a) MultiNet

(b) ResNet

(c) EfficientNet

(d) ViT

Figure 2: Confusion matrix for the ViT model on BreakHis dataset on the entire dataset including all magnifications 40X, 100X, 200X, 400X

Table 2shows the precision, recall, and F1-Score for the ViT model on the BreakHis dataset. ViT achieves 100% of performance for all the metrics on adenosis class. In addition, the model achieve attain 100% of recall on papillary carcinoma class.

Table 2: Precision, Recall, and F1-Score for the ViT model on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	1	1	1
DC	0.94	0.96	0.95
F	0.97	0.96	0.96
LC	0.77	0.68	0.73
MC	0.96	0.94	0.95
PC	0.92	1	0.96
PT	0.93	0.97	0.95
TA	0.98	0.95	0.96

2(a)and Table 3 display the performance of the ViT model integrated with MultiNet model. The model gain 100% of precision on adenosis class.

Table 3: Precision, Recall, and F1-Score for the ViT and MultiNet models on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	1	0.99	0.99
DC	0.94	0.95	0.95
F	0.97	0.95	0.96
LC	0.78	0.72	0.75
MC	0.98	0.98	0.98
PC	0.95	0.96	0.96
PT	0.96	0.97	0.96
TA	0.95	0.97	0.96

Precision, recall, and F1-score for the combined ViT with ResNet model on the BreakHis dataset are shown in Table 4.

Shown in 2(b) we can observe some extra miss-classification in the ViT with ResNet model compare to ViT with MultiNet model, 2(a).

Table 4: Precision, Recall, and F1-Score for the ViT and ResNet model on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.97	0.99	0.98
DC	0.94	0.96	0.95
F	0.94	0.93	0.93
LC	0.78	0.69	0.73
MC	0.97	0.97	0.97
PC	0.96	0.95	0.96
PT	0.93	0.96	0.94
TA	0.95	0.94	0.95

The ViT and EfficientNet combined model’s confusion matrix is presented in 2(c). Maximum accuracy of 98% is attained in the papillary carcinoma class.

In Table 5, we can see the precision, recall, and F1-score of the combined ViT and EfficientNet model on the BreakHis dataset. The highest value for precision is 99% and attained on two classes, adenosis and mucinous carcinoma. Furthermore, the model acquires the same performance of recall on papillary carcinoma class. 98% is the maximum value that the model reaches on F1-Score.

Table 5: Precision, Recall, and F1-Score for the ViT and EfficientNet model on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.99	0.97	0.98
DC	0.93	0.96	0.95
F	0.98	0.95	0.96
LC	0.79	0.69	0.74
MC	0.99	0.96	0.97
PC	0.98	0.99	0.98
PT	0.92	0.97	0.94
TA	0.94	0.97	0.95

4.2 DeiT-based models

Now, we will examine the performance of the DeiT-based model merged with other models, counting MultiNet, ResNet, and EfficientNet, utilizing assessment criteria such as confusion matrix, precision, recall, and F1-Score.

3(a)illustrates that the papillary carcinoma class has the highest accuracy at 98.8 percent. Except for lobular carcinoma class, all other groups have an accuracy rate of 94% or above.

(a) Deit

(b) ResNet

(c) EfficientNet

(d) MultiNet

(e) ViT

Figure 3: Confusion matrix for the Deit based models on BreakHis dataset on the entire dataset including all magnifications 40X, 100X, 200X, 400X

Table 6displays the precision, recall, and F1-score achieved by the DeiT model on the BreakHis dataset. Adenosis class yields the highest value, at 99%, for all the metrics. In addition, similar recall performance is achieved for the papillary carcinoma class.

Table 6: Precision, Recall, and F1-Score for the DeiT model on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.99	0.99	0.99
DC	0.94	0.95	0.95
F	0.95	0.97	0.96
LC	0.76	0.72	0.74
MC	0.97	0.95	0.96
PC	0.95	0.99	0.97
PT	0.97	0.94	0.96
TA	0.96	0.94	0.95

The confusion matrix for the combined model of DeiT with ResNet is depicted in 3(b). In the papillary carcinoma class, the highest possible degree of accuracy is achieved.

Table 7shows the performance of the DeiT model combined with the ResNet model using Precision, Recall, and F1-Score metrics. The top performance of these metrics all attain by the papillary carcinoma class with following values of 99%, 100%, and 99% for precision, recall and F1-Score respectively.

Table 7: Precision, Recall, and F1-Score for the DeiT and the ResNet models on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.98	0.96	0.97
DC	0.93	0.95	0.94
F	0.95	0.93	0.94
LC	0.73	0.69	0.71
MC	0.98	0.97	0.97
PC	0.99	1	0.99
PT	0.93	0.94	0.94
TA	0.94	0.94	0.94

In 3(c), we see the confusion matrix for the DeiT merged with the EfficientNet model, with accuracy at the highest attainable level in the papillary carcinoma class.

Table 8illustrates the overall performance of the DeiT with EfficientNet models with respect to the precision, recall, and F1-Score criteria. The adenosis class achieved the highest results across the board (99%) for each of these indicators of success. Papillary carcinoma class, on the other hand, is slightly easier for the model to identify (recall rate of 100%) than adenosis class(recall rate of 99%) and rest of the categories.

Table 8: Precision, Recall, and F1-Score for the DeiT and the EfficientNet models on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.99	0.99	0.99
DC	0.94	0.95	0.95
F	0.97	0.97	0.97
LC	0.74	0.71	0.72
MC	0.97	0.93	0.95
PC	0.98	1	0.99
PT	0.96	0.97	0.96
TA	0.95	0.94	0.95

3(d)presents the confusion matrix for the DeiT $\oplus$ MultiNet model. The maximum accuracy is observed in the class papillary carcinomas. In 6 out of 8 categories, the model has a success rate of 95% or above.

The DeiT with MultiNet model’s overall performance according to precision, recall, and F1-Score is shown in Table 9. Both the mucinous carcinoma class and the papillary carcinoma class have perfect precision and recall at 100%. In addition, it has a recall of 99% for the adenosis class. The F1-Score of the model is 98% across three different classes (adenosis, mucinous carcinoma, and the papillary carcinoma).

Table 9: Precision, Recall, and F1-Score for the DeiT and MultiNet models on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.97	0.99	0.98
DC	0.93	0.96	0.95
F	0.95	0.96	0.96
LC	0.78	0.67	0.72
MC	1	0.96	0.98
PC	0.97	1	0.98
PT	0.93	0.96	0.94
TA	0.98	0.94	0.96

3(e)depicts the confusion matrix for the DeiT paired with the ViT model, with the adenosis class achieving the best accuracy (97%) of all classes.

We also experimented with an ensemble model for Transformers, which combines the DeiT and ViT models into a unified framework for image classification. Table 10 displays the overall model results with regard to precision, recall, and F1-Score. The adenosis class achieves the highest possible score of 97% across all metrics.

Table 10: Precision, Recall, and F1-Score for the ViT and DeiT model on BreakHis dataset for the multiclass classification

Class	Precision	Recall	F1-Score
A	0.97	0.97	0.97
DC	0.92	0.95	0.94
F	0.95	0.93	0.94
LC	0.73	0.72	0.72
MC	0.96	0.90	0.93
PC	0.94	0.96	0.95
PT	0.91	0.91	0.91
TA	0.95	0.88	0.92

Figure 4displays the mean precision, recall, and F1-Score of all models utilized to conduct the experiments. As observed, MultiNet-ViT gets the best performance among all metrics at 94%. The proposed model outperforms all other transformer-based combinations. MultiNet-ViT offers a significant degree of generalizability for multiclass classification. Using multiple networks powered by transformers can result in a more reliable and efficient solution.

Figure 4: Average of the Precision, Recall and F1-Score of all the models for the multiclass classification on BreakHis dataset. Eff(EfficientNet)

5 Conclusion

Breast cancer is the most common type of cancer in females. Early detection and identification can stop the further development of breast cancer. The ideal application in automated breast cancer detection is still a work in progress. In this research, we propose a couple of novel Neural Network models to diagnose different cancer types. The architectures was tested on BreakHis dataset. To assure reliability, the proposed model has been evaluated using several metrics.

MultiNet-ViT is suggested to identify breast cancer on histological images. Taking into consideration both the global and local associations of images in a broad context, this approach combines the strengths of the traditional CNN model for extracting local information and the latest transformer model for collecting long-range correlation. The model was evaluated on the BreakHis dataset of breast cancer histopathology, where it showed an average of 94% precision, recall, and F1-Score. Results from our trials definitively show the transformer’s potential in the multiclass classification tasks of breast cancer sub-types. So as to guarantee the model’s generalization abilities, we have also expanded our detection goal to encompass all magnification factors. Finally, in order to gauge the efficacy of our model, we compared it to that of competing models. The proposed models improved the detection capability for different types of carcinomas, resulting in a more applicable scenario for clinical application. Our model is able to bridge the gap between transformer’s development and its limited use in medical imaging, particularly in breast cancer multiclass classification.

Transformers without transfer learning demand an excessive amount of data, training, and processing resources. Accordingly, it is recommended to utilize transfer learning, particularly in medical image processing. Future studies may find a similar strategy works well for other computer vision tasks, such as detection, segmentation, localization, etc. In addition, the architecture has not been applied to three-dimensional volumes, therefore any such future efforts would be quite informative.