[Bug]: in the Enhance's function Upscale Method

[Edwardsmith35]

Checklist

• The issue has not been resolved by following the troubleshooting guide
• The issue exists on a clean installation of Fooocus
• The issue exists in the current version of Fooocus
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

when generating a number of images while using the enhance's upscale function button, the prompts used during upscaling all images is the same prompt that was used to generate the first image, this coupled with a high denoising strength of 0.386 changes the image so much, potentially failing the upscale process and changing the original image after upscaling to a new one which is a combination of the first image and the second one.
Here's an example showcasing the the second image upscaling process of the three images I generated using a prompt:
animal in the wild

Steps to reproduce the problem

Use a wildcard containing a list of prompts, then generate the images while using the Enhance's upscale function to automatically upscale the images after generation, then looking at the log, when fooocus begins upscaling the second image, we notice that the prompt used during the upscaling process is the same one used to generate the first image, and this continues untill it finishes upscaling all the images.

What should have happened?

Fooocus should have used the original unique prompt that was used to generate the image during Enhance's automatic upscaling.

What browsers do you use to access Fooocus?

Google Chrome

Where are you running Fooocus?

Cloud (Google Colab)

What operating system are you using?

No response

Console logs

Collecting pygit2==1.15.1
  Downloading pygit2-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB)
Requirement already satisfied: cffi>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from pygit2==1.15.1) (1.17.1)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.16.0->pygit2==1.15.1) (2.22)
Downloading pygit2-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 48.0 MB/s eta 0:00:00
Installing collected packages: pygit2
  Attempting uninstall: pygit2
    Found existing installation: pygit2 1.16.0
    Uninstalling pygit2-1.16.0:
      Successfully uninstalled pygit2-1.16.0
Successfully installed pygit2-1.15.1
/content
Cloning into 'Fooocus'...
remote: Enumerating objects: 6718, done.
remote: Total 6718 (delta 0), reused 0 (delta 0), pack-reused 6718 (from 1)
Receiving objects: 100% (6718/6718), 33.25 MiB | 26.25 MiB/s, done.
Resolving deltas: 100% (3873/3873), done.
/content/Fooocus
Already up-to-date
Update succeeded.
[System ARGV] ['entry_with_update.py', '--share', '--always-high-vram']
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]
Fooocus version: 2.5.5
Error checking version for torchsde: No package metadata was found for torchsde
Installing requirements
[Cleanup] Attempting to delete content of temp dir /tmp/fooocus
[Cleanup] Cleanup successful
Downloading: "https://huggingface.co/lllyasviel/misc/resolve/main/xlvaeapp.pth" to /content/Fooocus/models/vae_approx/xlvaeapp.pth

100% 209k/209k [00:00<00:00, 13.6MB/s]
Downloading: "https://huggingface.co/lllyasviel/misc/resolve/main/vaeapp_sd15.pt" to /content/Fooocus/models/vae_approx/vaeapp_sd15.pth

100% 209k/209k [00:00<00:00, 10.6MB/s]
Downloading: "https://huggingface.co/mashb1t/misc/resolve/main/xl-to-v1_interposer-v4.0.safetensors" to /content/Fooocus/models/vae_approx/xl-to-v1_interposer-v4.0.safetensors

100% 5.40M/5.40M [00:00<00:00, 44.7MB/s]
Downloading: "https://huggingface.co/lllyasviel/misc/resolve/main/fooocus_expansion.bin" to /content/Fooocus/models/prompt_expansion/fooocus_expansion/pytorch_model.bin

100% 335M/335M [00:01<00:00, 237MB/s]
Downloading: "https://huggingface.co/lllyasviel/fav_models/resolve/main/fav/juggernautXL_v8Rundiffusion.safetensors" to /content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors

100% 6.62G/6.62G [00:32<00:00, 216MB/s]
Downloading: "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_offset_example-lora_1.0.safetensors" to /content/Fooocus/models/loras/sd_xl_offset_example-lora_1.0.safetensors

100% 47.3M/47.3M [00:00<00:00, 197MB/s]
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: HIGH_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
Refiner unloaded.
IMPORTANT: You are using gradio version 3.41.2, however version 4.44.1 is available, please upgrade.
--------
Running on local URL:  http://127.0.0.1:7865
Running on public URL: https://25bca3e66e577e9417.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
loaded straight to GPU
Requested to load SDXL
Loading 1 new model
Base model loaded: /content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
VAE loaded: None
Request to load LoRAs [('sd_xl_offset_example-lora_1.0.safetensors', 0.1)] for model [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.58 seconds
2024-12-13 15:04:20.460856: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-12-13 15:04:20.478013: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-12-13 15:04:20.482969: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-12-13 15:04:20.495543: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-12-13 15:04:22.418894: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Started worker with PID 948
App started successful. Use the app with http://127.0.0.1:7865/ or 127.0.0.1:7865 or https://25bca3e66e577e9417.gradio.live
[Parameters] Adaptive CFG = 7
[Parameters] CLIP Skip = 2
[Parameters] Sharpness = 2
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] Seed = 5105782782636255632
[Parameters] CFG = 4
[Fooocus] Loading control models ...
[Parameters] Sampler = dpmpp_2m_sde_gpu - karras
[Parameters] Steps = 30 - 15
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Refiner unloaded.
[Fooocus] Processing prompts ...
[Wildcards] processing: __animal__ in the wild
[Wildcards] Alligator in the wild
[Wildcards] processing: __animal__ in the wild
[Wildcards] Ant in the wild
[Wildcards] processing: __animal__ in the wild
[Wildcards] Antelope in the wild
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] Alligator in the wild, handsome, intricate, elegant, highly detailed, colorful, vivid color, romantic, sharp focus, extremely fine detail, cinematic, singular, dramatic shining, symmetry, illuminated, epic, brilliant, light, coherent, calm, vibrant, glowing, complex, amazing composition, decorative, elite, elaborate, iconic, imposing, rich deep colors, peaceful
[Fooocus] Preparing Fooocus text #2 ...
[Prompt Expansion] Ant in the wild, inspired, highly detailed, intricate, sharp focus, cinematic light, new epic great creative hopeful, dramatic ambient, dynamic, shiny, vibrant, incredible composition, magical atmosphere, professional fine detail, clear, neat, beautiful, stunning, pretty, attractive, delicate, complex, glowing, fantastic, elegant, fancy, inspiring, brilliant, amazing, awesome
[Fooocus] Preparing Fooocus text #3 ...
[Prompt Expansion] Antelope in the wild, highly detailed, sharp focus, beautiful, intricate, elegant, confident, dynamic, vibrant colors, cinematic, fine, sublime, inspired, rich deep color, surreal, light, ambient background, radiant, amazing, symmetry, illuminated, pretty, excellent composition, crisp, extremely complex, epic, artistic,, appealing, full detail, striking, iconic
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.14 seconds
[Fooocus] Encoding positive #2 ...
[Fooocus] Encoding positive #3 ...
[Fooocus] Encoding negative #1 ...
[Fooocus] Encoding negative #2 ...
[Fooocus] Encoding negative #3 ...
[Parameters] Denoising Strength = 1.0
[Parameters] Initial Latent shape: Image Space (896, 1152)
Preparation time: 4.98 seconds
Using karras scheduler.
[Fooocus] Preparing task 1/3 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 2.12 seconds
100% 30/30 [00:25<00:00,  1.20it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.34 seconds
[Fooocus] Saving image 1/3 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
Generating and saving time: 30.24 seconds
[Fooocus] Preparing task 2/3 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.72 seconds
100% 30/30 [00:25<00:00,  1.19it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.29 seconds
[Fooocus] Saving image 2/3 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
Generating and saving time: 28.87 seconds
[Fooocus] Preparing task 3/3 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.71 seconds
100% 30/30 [00:25<00:00,  1.19it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.27 seconds
[Fooocus] Saving image 3/3 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
Generating and saving time: 28.90 seconds
[Fooocus] Processing enhance ...
[Fooocus] Downloading upscale models ...
Downloading: "https://huggingface.co/lllyasviel/misc/resolve/main/fooocus_upscaler_s409985e5.bin" to /content/Fooocus/models/upscale_models/fooocus_upscaler_s409985e5.bin

100% 32.1M/32.1M [00:00<00:00, 192MB/s]
[Fooocus] Upscaling image from (1152, 896) ...
Upscaling image with shape (896, 1152, 3) ...
Image upscaled.
[Fooocus] VAE encoding ...
Final resolution is (2304, 1792).
[Fooocus] Preparing enhance prompts ...
[Fooocus] Loading models ...
Refiner unloaded.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.88 seconds
[Fooocus] Processing prompts ...
[Wildcards] processing: __animal__ in the wild
[Wildcards] Alligator in the wild
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] Alligator in the wild, handsome, intricate, elegant, highly detailed, colorful, vivid color, romantic, sharp focus, extremely fine detail, cinematic, singular, dramatic shining, symmetry, illuminated, epic, brilliant, light, coherent, calm, vibrant, glowing, complex, amazing composition, decorative, elite, elaborate, iconic, imposing, rich deep colors, peaceful
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 0.5689725279808044
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.64 seconds
100% 18/18 [01:22<00:00,  4.59s/it]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.29 seconds
[Fooocus] Saving image 1/6 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
[Fooocus] Preparing enhancement 2/6 ...
[Enhance] Searching for "face"
Downloading: "https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth" to /content/Fooocus/models/inpaint/groundingdino_swint_ogc.pth

100% 662M/662M [00:05<00:00, 137MB/s]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:534: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3595.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
tokenizer_config.json: 100% 48.0/48.0 [00:00<00:00, 323kB/s]
config.json: 100% 570/570 [00:00<00:00, 3.43MB/s]
vocab.txt: 100% 232k/232k [00:00<00:00, 3.61MB/s]
tokenizer.json: 100% 466k/466k [00:00<00:00, 3.33MB/s]
model.safetensors: 100% 440M/440M [00:06<00:00, 71.5MB/s]
/content/Fooocus/modules/patch.py:465: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  result = original_loader(*args, **kwargs)
Requested to load GroundingDINO
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.36 seconds
/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:1060: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py:632: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/groundingdino/models/GroundingDINO/transformer.py:862: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
Downloading: "https://huggingface.co/mashb1t/misc/resolve/main/sam_vit_b_01ec64.pth" to /content/Fooocus/models/sam/sam_vit_b_01ec64.pth

100% 358M/358M [00:08<00:00, 42.4MB/s]
Requested to load Sam
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.44 seconds
[Enhance] 1 boxes detected
[Enhance] 1 segments detected in boxes
[Enhance] 1 segments applied to mask
[Fooocus] Preparing enhance prompts ...
[Fooocus] Loading models ...
Refiner unloaded.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.85 seconds
[Fooocus] Processing prompts ...
[Wildcards] processing: __animal__ in the wild
[Wildcards] Alligator in the wild
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] Alligator in the wild, handsome, intricate, elegant, highly detailed, colorful, vivid color, romantic, sharp focus, extremely fine detail, cinematic, singular, dramatic shining, symmetry, illuminated, epic, brilliant, light, coherent, calm, vibrant, glowing, complex, amazing composition, decorative, elite, elaborate, iconic, imposing, rich deep colors, peaceful
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
[Fooocus] VAE Inpaint encoding ...
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.21 seconds
[Fooocus] VAE encoding ...
Final resolution is (2304, 1792), latent is (1024, 1024).
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 0.5958056449890137
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.61 seconds
100% 30/30 [00:25<00:00,  1.16it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.30 seconds
[Fooocus] Saving image 2/6 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
Enhancement time: 65.81 seconds
Enhancement image time: 233.97 seconds
[Fooocus] Downloading upscale models ...
[Fooocus] Upscaling image from (1152, 896) ...
Upscaling image with shape (896, 1152, 3) ...
Image upscaled.
[Fooocus] VAE encoding ...
Final resolution is (2304, 1792).
[Fooocus] Preparing enhance prompts ...
[Fooocus] Loading models ...
Refiner unloaded.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.90 seconds
[Fooocus] Processing prompts ...
[Wildcards] processing: __animal__ in the wild
[Wildcards] Alligator in the wild
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] Alligator in the wild, handsome, intricate, elegant, highly detailed, colorful, vivid color, romantic, sharp focus, extremely fine detail, cinematic, singular, dramatic shining, symmetry, illuminated, epic, brilliant, light, coherent, calm, vibrant, glowing, complex, amazing composition, decorative, elite, elaborate, iconic, imposing, rich deep colors, peaceful
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 0.5689725279808044
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.28 seconds
100% 18/18 [01:22<00:00,  4.60s/it]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.28 seconds
[Fooocus] Saving image 3/6 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-12-13/log.html
[Fooocus] Preparing enhancement 4/6 ...
[Enhance] Searching for "face"
Requested to load GroundingDINO
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.39 seconds
Requested to load Sam
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.47 seconds
[Enhance] 1 boxes detected
[Enhance] 1 segments detected in boxes
[Enhance] 1 segments applied to mask
[Fooocus] Preparing enhance prompts ...
[Fooocus] Loading models ...
Refiner unloaded.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.80 seconds
[Fooocus] Processing prompts ...
[Wildcards] processing: __animal__ in the wild
[Wildcards] Alligator in the wild
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] Alligator in the wild, handsome, intricate, elegant, highly detailed, colorful, vivid color, romantic, sharp focus, extremely fine detail, cinematic, singular, dramatic shining, symmetry, illuminated, epic, brilliant, light, coherent, calm, vibrant, glowing, complex, amazing composition, decorative, elite, elaborate, iconic, imposing, rich deep colors, peaceful
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
[Fooocus] VAE Inpaint encoding ...
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.90 seconds
[Fooocus] VAE encoding ...
Final resolution is (2304, 1792), latent is (1152, 896

Additional information

No response

[Edwardsmith35]

I managed to fixed the bug, here's a link to my fork:
https://github.com/Edwardsmith35/Fooocus

[Edwardsmith35]

I managed to fix the issue, and other issues that I stumbled upon.

Here's the link to my Fork:
https://github.com/Edwardsmith35/Fooocus-Enhanced