Feat/batch predict age and gender

[NatLee]

Tickets

#441
#678
#1069
#1101

What has been done

With this PR, new predicts function to support batch predictions.

How to test

Use this class, user can load model and predict in batch.

class AgeGenderModel():
    """
    Age and gender model
    """
    def __init__(self):
        self.target_size = (224, 224)
        self.age_model, self.gender_model = self.load()

    def load(self) -> Tuple:
        age_model = modeling.build_model(task="facial_attribute", model_name="Age")
        gender_model = modeling.build_model(task="facial_attribute", model_name="Gender")
        return age_model, gender_model

    def process_data(self, data: np.ndarray) -> np.ndarray:
        """
        Process input image data
        """
        img_content = data[:, :, ::-1] # rgb to bgr
        img_content = preprocessing.resize_image(
            img=img_content,
            target_size=self.target_size
        )
        return img_content

    def predict(self, data: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
        """
        Predict age and gender for one input image
        """
        img_content = self.process_data(data)
        age = self.age_model.predict(img_content)
        gender = self.gender_model.predict(img_content)
        return gender, age

    def predicts(self, data: List[np.ndarray]) -> Tuple[List, List]:
        """
        Predict age and gender for batch input images
        """
        img_contents = [self.process_data(img_content) for img_content in data]
        ages = self.age_model.predicts(img_contents)
        genders = self.gender_model.predicts(img_contents)
        return genders, ages 

Time Costs for 10 Images:

Original for-loop prediction: 1.91108 seconds

Loading split Age and Gender models: 0.00002 seconds
Batch prediction: 0.58243 seconds
Single prediction: 0.03749 seconds

Here's my test script:

from deepface import DeepFace
import cv2
import time
# Load one face
img = cv2.imread("./test.png")

# Make it as a list
imgs = [img] * 10

# For-loop predict
start = time.time()
print("=====For-loop Predict=====")
for img in imgs:
    objs = DeepFace.analyze(
    img_path=img,
    actions=['age', 'gender'],
    )
print(f"Time: {time.time() - start:.5f}s")

# Load models
print("=====Load Split Models=====")
start = time.time()
model = AgeGenderModel()
print(f"Time: {time.time() - start:.5f}s")

# Batch Predict
start = time.time()
genders, ages = model.predicts(imgs)
print("=====Batch Predict=====")
print(genders, ages)
print(f"Time: {time.time() - start:.5f}s")

# Single Predict
start = time.time()
gender, age = model.predict(img)
print("=====Single Predict=====")
print(gender, age)
print(f"Time: {time.time() - start:.5f}s")

predicts is placed in age and gender clients to keep the DeepFace.analyze function logic.

Thank you for taking time to go through this feedback. :)

[h-alice]

Bump
Really needs this feature.

[serengil]

I don't support having another predicts function. Instead, you can add that logic under predict.

1- predict accepts both single image and list of images as

img: Union[np.ndarray, List[np.ndarray]]

2- in predict function, you can check the type of img, and redirect it to your logic if it is list as

if isinstance(img, np.ndarray):
   # put old predict logic here
elif isinstance(img, np.ndarray):
   # put your batch processing logic here

3- this new logic is worth to have its own unit tests. possibly, you can add some unit tests here.

4- return type of predict should be Union[np.float64, np.ndarray]

5- You should also update the interface in DeepFace.py

[serengil]

Actions failed because of linting - link

************* Module deepface.models.demography.Age
pylint: Command line or configuration file:1: UserWarning: 'Exception' is not a proper value for the 'overgeneral-exceptions' option. Use fully qualified name (maybe 'builtins.Exception' ?) instead. This will cease to be checked at runtime in 3.1.0.
deepface/models/demography/Age.py:70:0: C0303: Trailing whitespace (trailing-whitespace)
************* Module deepface.models.demography.Emotion
deepface/models/demography/Emotion.py:88:0: C0303: Trailing whitespace (trailing-whitespace)

[serengil]

model.predict causes memory issue when it is called in a for loop, that is why we call it as self.model(img, training=False).numpy()[0, :]

in your design, if this is called in a for loop, still it will cause memory problem.

IMO, if it is single image, we should call self.model(img, training=False).numpy()[0, :], it is many faces then call self.model.predict_on_batch

[h-alice]

Thank you for sharing your perspective on this matter.

We found the issue you mentioned is also mentioned in this page: tensorflow/tensorflow#44711. We believe it’s being resolved.

Furthermore, if we can utilize the batch prediction method provided in this PR, we may be able to avoid repeatedly calling the predict function within a loop of unrolled batch images, which is the root cause of the memory issue you described.

We recommend that you consider retaining our batch prediction method.

[serengil]

hey, even though this is sorted in newer tf versions, many users using old tf versions raise tickets about this problem. so, we should consider the people using older tf version. that is why, i recommend to use self.model(img, training=False).numpy()[0, :] for single images, and self.model.predict_on_batch for batches.

[h-alice]

Hi! 👋
Please take a look at our prediction function, which uses the legacy single prediction method you suggested, and also provides batch prediction if a batch of images is provided.

Please let us know if there’s anything else we can improve. Any advice you have is greatly appreciated.

deepface/deepface/models/Demography.py

Line 24 in a23893a

def _predict_internal(self, img_batch: np.ndarray) -> np.ndarray:

[serengil]

hconcat makes a single image

input image shape before hconcat is (1728, 2500, 3)
input image shape after hconcat is (1728, 5000, 3)

to have a numpy array with (2, 1728, 2500, 3) shape, you should do something like:

img = np.stack((img, img), axis=0)

[serengil]

Also please check that img is now having (2, x, x, x) shape

[serengil]

Finally, unit tests failed for that input. The case you tested did not test what you did. It is still single image.

[h-alice]

Hi @serengil 👋
We have implemented batched images support in DeepFace::analysis, and the test cases have been modified as per your request. Please help us check if this matches your requirements.

Due to the complexity of designing a more efficient flow for analysis, we will prioritize extending the functionality of models that can accept batched images for now.

We will discuss enhancing the performance of the analysis function in separate threads or through pull requests. We would invite you to participate in these discussions once we are ready.

Please help us merge this PR if all the requirements are met.

[serengil]

i initially confused about why why squeeze first and expand dimensions second

would you please add a comment here something like:

we did perform squeeze and expand dimensions sequentially to have same behaviour for (224, 224, 3), (1, 224, 224, 3) and (n, 224, 224, 3)

[h-alice]

We took a look at the processing flow and discovered that the squeeze operation is unnecessary. Every single image input would have an expanded batch dimension of (1, 224, 224, 3), so there’s no need to handle inputs with this dimension.

The redundant squeeze process has been removed.

[serengil]

I am not available to review it until early Feb.

[serengil]

Tests failed