Machine Learning Project – Pnuemonia Detection using CNN in Python

Pneumonia is a dangerous lung infection that’s hard to diagnose in its early stages. Fortunately, with the help of deep learning and AI, we can detect it accurately from medical images like X-rays and CT scans.

At Pythongeeks, we are going to develop a model trained on a large dataset of medical images that uses neural networks to detect pneumonia with high accuracy. By using our model, doctors can diagnose pneumonia faster and more accurately, improving patient outcomes and reducing workload. Our goal is to use the latest advancements in AI to improve healthcare and make a positive impact on people’s lives.

What is Deep Learning?

Deep learning is a type of AI that uses neural networks inspired by the human brain to recognize complex patterns in data. It’s trained using large amounts of data and can be applied to a wide range of applications, including image and speech recognition, natural language processing, and more. Deep learning has shown great potential in areas like medical diagnosis and climate modeling and is set to become even more powerful as data and computing power continue to increase.

Dataset Preparation

Dataset preparation is a critical step in developing a model for pneumonia detection. The quality and quantity of the data used to train the model can have a significant impact on the accuracy and effectiveness of the model.
To develop a pneumonia detection model we are using the Chest X-ray images dataset available on kaggle containing the X-ray images of NORMAL and PNEUMONIA. In this dataset, images are classified into NORMAL and PNEUMONIA classes. We are training our model using a training and testing set of X-ray images and doing the validation of model using a validation set of X-ray images.

Healthy Lung X-ray

Pneumonia Lung X-ray

The red arrows in Pneumonia Lung indicate white infiltrates, a distinguishing feature of pneumonia.

Prerequisites for Machine Learning Pneumonia Detection

To develop a deep learning model for pneumonia detection, you need to have a good understanding of deep learning principles and techniques, as well as experience with Python and its libraries such as NumPy, Pandas, TensorFlow, and Keras. Additionally, knowledge of medical imaging, pneumonia symptoms and causes, and access to a large dataset of medical images are necessary. You should also be familiar with data preprocessing techniques, model development and training, evaluation and testing, and deployment of deep learning models in a production environment.

1. Python 3.7 and above
2. Any python editor (Suggested Jupyter Notebook)

Download Machine Learning Pneumonia Detection Project

Please download the source code of Machine Learning Pneumonia Detection Project: Machine Learning Pneumonia Detection Project Code.

Steps to Create Machine Learning Pneumonia Detection Project

Following are the steps for developing the Machine Learning Pneumonia Detection Project:

Step 1: Installation
Step 2: Model development
Step 3: Deployment

Installation

Open windows cmd as administrator

1. To install the numpy library run the command from the cmd.

pip install numpy

2. To install tensorflow run the command from cmd.

pip install tensorflow

Model development

We have the dataset of chest x-ray. By using deep learning, we train our model on this dataset. Dataset basically has two classes, ‘NORMAL’ and ‘PNEUMONIA’.
Let’s start.

1. First of all, we are importing all the necessary libraries that are required during the development of our model.

import os
import numpy as np
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout
from tensorflow.keras.models import Sequential

2. These are the paths to the directories containing the training, testing, and validation datasets of images, respectively.

train_data = 'Dataset/train'
test_data = 'Dataset/test'
val_data = 'Dataset/val'

3. This initializes a tool called ImageDataGenerator that generates batches of image data for training a deep learning model. The tool applies different image transformations such as rescaling, shear, zoom, and horizontal flip to create new versions of the original images. These transformations help the model learn to recognize features in images more effectively and prevent overfitting, which is when the model performs well on the training data but poorly on new data.

train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2,
                                   horizontal_flip=True)

4. This code initializes an image data generator for testing or validation purposes. The generator rescales the pixel values of the images to be in the range of [0, 1]. This normalization step ensures that the testing or validation data is processed in the same way as the training data, which is important for accurate model evaluation.

test_datagen = ImageDataGenerator(rescale=1./255)

5. These lines of code create generators that prepare the image data for training, testing, and validation of the deep learning model. They resize the images to a specific size, group them into batches, and specify the number of classes (in this case, two classes). The generators also apply image augmentation techniques and convert the images into tensors that the deep learning model can use during training, testing, and validation.

train_generator = train_datagen.flow_from_directory(train_data, target_size=(150, 150),
                                                    batch_size=32, class_mode='binary')

test_generator = test_datagen.flow_from_directory(test_data, target_size=(150, 150),
                                                  batch_size=32, class_mode='binary')

val_generator = test_datagen.flow_from_directory(val_data, target_size=(150, 150),
                                                 batch_size=32, class_mode='binary')

6. It initializes a sequential model using the Sequential class from the tensorflow.keras.models module.

model = Sequential()

7. This creates a deep learning model with different types of layers, including convolutional, pooling, flatten, dense, and dropout layers. These layers process and transform the input image data to learn features and classify images as either positive or negative for pneumonia.

model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

8. The code compiles the sequential model with a specified loss function, optimizer, and evaluation metrics. The loss function used here is binary_crossentropy, which is commonly used for binary classification problems. The optimizer used is RMSprop, with a learning rate of 1e-4. The evaluation metrics specified are accuracy, which is the percentage of correctly classified images, and binary cross-entropy loss.

model.compile(loss='binary_crossentropy', optimizer=tf.keras.optimizers.RMSprop(lr=1e-4), metrics=['acc'])

9. The training data is provided through the train_generator, and the validation data is provided through the val_generator. The model is trained for 20 epochs. The fit method returns a history object that contains information about the training and validation loss and accuracy for each epoch.

history = model.fit(train_generator, epochs=20, validation_data=val_generator)

It will take a lot of time to train, depending on your system configuration. Increasing the number of epochs can improve the model’s performance on the training data, but too many epochs can cause overfitting. Too few epochs can lead to underfitting, where the model hasn’t learned enough. It’s important to find the right number of epochs to balance between overfitting and underfitting.

10. Evaluate how well our trained model performs on a set of new images called the test dataset. It calculates the accuracy and loss of the model on the test data and saves the results to variables. The print statement displays the accuracy of the model on the test data.

test_loss, test_acc = model.evaluate(test_generator)
print('Test accuracy:', test_acc)

11. At the end, we save our model in the main folder with the name “pneumonia_detection_model.h5”.

model.save('pneumonia_detection_model.h5')

Deployment

After training a deep learning model to detect pneumonia from chest X-ray images, the code provided saves the model as a h5 file on the local system. This saved model can then be loaded and used to make predictions on new, unseen chest X-ray images. This is done by passing the images through the saved model and using its output to make a binary classification prediction (pneumonia or not). By doing so, the model can be used for real-world applications, such as assisting radiologists in detecting pneumonia in patients.

1. First of all we are importing all the necessary libraries that are required during deployment of our model and ignoring the warnings.

import warnings
warnings.filterwarnings('ignore')
import cv2
from tensorflow.keras.preprocessing.image import load_img, img_to_array
import numpy as np
import tensorflow as tf

2. Set a variable called “img_path” to the file path of an X-ray image that wants to test.

img_path = 'Dataset/val/NORMAL/NORMAL2-IM-1427-0001.jpeg'

3. Load an image from the specified file path, resize it to (150, 150) and convert it into a 4D NumPy array. The pixel values of the image are scaled down to be between 0 and 1.0. This is necessary because the trained model expects input data in this format. The shape of the 4D array is (1, 150, 150, 3), where the first dimension represents the number of images, the second and third dimensions represent the height and width of the image, and the fourth dimension represents the number of channels, which is 3 for color images.

img = load_img(img_path, target_size=(150, 150))
img_array = img_to_array(img)
img_array = img_array / 255.0 
img_array = np.expand_dims(img_array, axis=0) 

4. Load our trained model.

model = tf.keras.models.load_model('pneumonia_detection_model.h5')

5. By using our trained model, we are making prediction on a given image. Prediction contains a value between 0 and 1.

prediction = model.predict(img_array)

6. If the prediction value returned by the model is less than 0.5, it means that the input X-ray image is likely of a normal person. On the other hand, if the prediction value is greater than 0.5, it indicates that the input X-ray image is likely of a person affected by pneumonia. This is because the model was trained to classify input images as either normal or pneumonia, and a prediction value greater than 0.5 indicates a higher probability of the input image belonging to the pneumonia class.

if prediction < 0.5:
    print("This image does not contain pneumonia.")
else:
    print("This image contains pneumonia.")

This way we train a model using a labeled dataset to learn patterns and make predictions on unseen data. Once we have trained a model, we can save it to a file for future use. During deployment, we load the saved model and use it to make predictions on new data. This allows us to apply our trained model to real-world problems and make predictions on unseen data.

Conclusion

Deep learning technology has shown potential in accurately detecting pneumonia from medical images such as X-rays and CT scans, which can improve diagnoses and treatment outcomes. However, further research is needed to improve its accuracy and safety. The ethical and legal issues of using deep learning in medical applications should also be considered. Nonetheless, this promising technology has the potential to revolutionize pneumonia diagnosis and treatment and will continue to be researched in the future.