[BUGFIX] [issue#17634] Directly delete the /var/.regenerate file dire…

[lewisvoncken]

…ctly after checking it it exists

= cleanGeneratedFiles is called during the create function of the ObjectManager which is called during cronjobs

Description (*)

Fix according to the comment of @adrian-martinez-interactiv4
#17634 (comment)

other solution will be removing cleanGeneratedFiles from \Magento\Framework\App\ObjectManagerFactory::create

Fixed Issues (if relevant)

1. Cache disabled after run "composer update" command #17634: Cache disabled after run "composer update" command

Manual testing scenarios (*)

1. Cache disabled after run "composer update" command #17634 (comment)
2. or run composer install + cron in background

Questions or comments

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @lewisvoncken. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.3-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Guide documentation.

[magento-engcom-team]

Hi @kalpmehta, thank you for the review.
ENGCOM-5831 has been created to process this Pull Request
✳️ @kalpmehta, could you please add one of the following labels to the Pull Request?

Label	Description
Auto-Tests: Covered	All changes in Pull Request is covered by auto-tests
Auto-Tests: Not Covered	Changes in Pull Request requires coverage by auto-tests
Auto-Tests: Not Required	Changes in Pull Request does not require coverage by auto-tests

[ihor-sviziev]

Hi @lewisvoncken @kalpmehta @adrian-martinez-interactiv4,
I just reviewed this part and didn’t get why we need to disable cache at all when removing cache directory? I think it should be safe enough just remove disabling cache.

[lewisvoncken]

Hi @lewisvoncken @kalpmehta @adrian-martinez-interactiv4,
I just reviewed this part and didn’t get why we need to disable cache at all when removing cache directory? I think it should be safe enough just remove disabling cache.

Hi @ihor-sviziev
If the others agree on that solution I will update the PR.

[maaarghk]

Maybe this info can help you decide - https://gist.github.com/piotrekkaminski/bb3967a666ae03ca7d9766252d19376e

[lewisvoncken]

I think that this solution isn't changing the working but only makes sure the regeneration process isn't started twice.

[kalpmehta]

@ihor-sviziev I agree with you. May be the system would not allow to delete the cache directories when there is active file writing in progress? I am not sure.

However, this PR is not changing any such behavior and simply fixes the existing bug reported in #17634 so I didn't put much attention to the bigger picture.

[ihor-sviziev]

Hi @joni-jones,
Could you review this PR and add your feedback? As for me - we could remove disabling cache at all.

[adrian-martinez-interactiv4]

Hi @ihor-sviziev @kalpmehta @lewisvoncken , I've just seen this PR, sorry for the delay.

About removing disabling cache, I agree, as it is not needed by any class involved in cleaning generated files process, so it can be removed along with accesory private methods not needed anymore (getEnabledCacheTypes, disableAllCacheTypes, enableCacheTypes, getEnvPath).

But caches remaining disabled is not the problem itself, but the consequence of a filesystem error when concurrence occurs while trying to clean generated code (https://youtu.be/9-X1cIIY7y8).

The easiest implementation is putting up the deletion of regenerate flag as follows (current + proposed cache disable removal):

<?php
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
namespace Magento\Framework\Code;

use Magento\Framework\App\Filesystem\DirectoryList;
use Magento\Framework\Filesystem\Directory\WriteFactory;
use Magento\Framework\Filesystem\Directory\WriteInterface;

/**
 * Clean generated code, DI configuration and cache folders
 */
class GeneratedFiles
{
    /**
     * Regenerate flag file name
     */
    const REGENERATE_FLAG = '/var/.regenerate';

    /**
     * @var DirectoryList
     */
    private $directoryList;

    /**
     * @var WriteInterface
     */
    private $write;

    /**
     * Constructor
     *
     * @param DirectoryList $directoryList
     * @param WriteFactory $writeFactory
     */
    public function __construct(DirectoryList $directoryList, WriteFactory $writeFactory)
    {
        $this->directoryList = $directoryList;
        $this->write = $writeFactory->create(BP);
    }

    /**
     * Clean generated code and DI configuration
     *
     * @return void
     *
     * @deprecated 100.1.0
     * @see \Magento\Framework\Code\GeneratedFiles::cleanGeneratedFiles
     */
    public function regenerate()
    {
        $this->cleanGeneratedFiles();
    }

    /**
     * Clean generated/code, generated/metadata and var/cache
     *
     * @return void
     */
    public function cleanGeneratedFiles()
    {
        if ($this->write->isExist(self::REGENERATE_FLAG)) {
            $this->write->delete(self::REGENERATE_FLAG);

            // Clean generated/code dir
            $this->deletePathCodeFolder(DirectoryList::GENERATED_CODE);
            $this->deletePathCodeFolder(DirectoryList::GENERATED_METADATA);
            $this->deletePathCodeFolder(DirectoryList::CACHE);
        }
    }

    /**
     * Create flag for cleaning up generated content
     *
     * @return void
     */
    public function requestRegeneration()
    {
        $this->write->touch(self::REGENERATE_FLAG);
    }

    /**
     * @param string $pathType
     */
    private function deletePathCodeFolder(string $pathType)
    {
        $relativePath = $this->write->getRelativePath($this->directoryList->getPath($pathType));
        if ($this->write->isDirectory($relativePath)) {
            $this->write->delete($relativePath);
        }
    }
}

Advantages

• Simple, prevents double execution
• BC

Disadvantages

• It does not ensure cleaning up has been properly performed, as it first removes its own flag and folder removing may fail, so not-deleted generated files won't be cleaned by next processes because of missing flag.
• It does not prevent processes execution while cleaning up is in progress, so it may lead to wrong executions or folder removing errors, due to concurrency of folder removal and folder writing by other processes.

I propose another one, using file locking and concurrency error control, modifying two files:

Class \Magento\Framework\Code\GeneratedFiles:

<?php
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */

namespace Magento\Framework\Code;

use Magento\Framework\App\Filesystem\DirectoryList;
use Magento\Framework\Exception\FileSystemException;
use Magento\Framework\Exception\RuntimeException;
use Magento\Framework\Filesystem\Directory\WriteFactory;
use Magento\Framework\Filesystem\Directory\WriteInterface;
use Magento\Framework\Lock\LockManagerInterface;

/**
 * Clean generated code, DI configuration and cache folders
 */
class GeneratedFiles
{
    /**
     * Regenerate flag file name
     */
    const REGENERATE_FLAG = '/var/.regenerate';

    /**
     * Regenerate lock file name
     */
    const REGENERATE_LOCK = self::REGENERATE_FLAG . '.lock';

    /**
     * Acquire regenerate lock timeout
     */
    const REGENERATE_LOCK_TIMEOUT = 5;

    /**
     * @var DirectoryList
     */
    private $directoryList;

    /**
     * @var WriteInterface
     */
    private $write;

    /**
     * @var LockManagerInterface
     */
    private $lockManager;

    /**
     * GeneratedFiles constructor.
     *
     * @param DirectoryList $directoryList
     * @param WriteFactory $writeFactory
     * @param LockManagerInterface $lockManager
     */
    public function __construct(
        DirectoryList $directoryList,
        WriteFactory $writeFactory,
        LockManagerInterface $lockManager
    ) {
        $this->directoryList = $directoryList;
        $this->write = $writeFactory->create(BP);
        $this->lockManager = $lockManager;
    }

    /**
     * Clean generated code and DI configuration
     *
     * @return void
     *
     * @deprecated 100.1.0
     * @see \Magento\Framework\Code\GeneratedFiles::cleanGeneratedFiles
     */
    public function regenerate()
    {
        $this->cleanGeneratedFiles();
    }

    /**
     * Clean generated/code, generated/metadata and var/cache
     * @return void
     */
    public function cleanGeneratedFiles()
    {
        if ($this->isCleanGeneratedFilesAllowed() && $this->acquireLock()) {
            try {
                $this->deleteFolder(DirectoryList::GENERATED_CODE);
                $this->deleteFolder(DirectoryList::GENERATED_METADATA);
                $this->deleteFolder(DirectoryList::CACHE);
                $this->write->delete(self::REGENERATE_FLAG);
            } catch (FileSystemException $exception) {
                // A filesystem error occured, possible concurrency error while trying
                // to delete a generated folder being used by another process.
                // Request regeneration for the next and unlock
                $this->requestRegeneration();
            } finally {
                $this->lockManager->unlock(self::REGENERATE_LOCK);
            }
        }
    }

    /**
     * Create flag for cleaning up generated content
     *
     * @return void
     */
    public function requestRegeneration()
    {
        $this->write->touch(self::REGENERATE_FLAG);
    }

    /**
     * Clean generated files is allowed if requested and not locked
     *
     * @return bool
     */
    private function isCleanGeneratedFilesAllowed(): bool
    {
        try {
            $isAllowed = $this->write->isExist(self::REGENERATE_FLAG)
                && !$this->lockManager->isLocked(self::REGENERATE_LOCK);
        } catch (FileSystemException | RuntimeException $e) {
            // Possible filesystem problem
            $isAllowed = false;
        }

        return $isAllowed;
    }

    /**
     * @return bool
     */
    private function acquireLock(): bool
    {
        try {
            $lockAcquired = $this->lockManager->lock(self::REGENERATE_LOCK, self::REGENERATE_LOCK_TIMEOUT);
        } catch (RuntimeException $exception) {
            // Lock not acquired due to FS problem
            $lockAcquired = false;
        }

        return $lockAcquired;
    }

    /**
     * @param string $pathType
     */
    private function deleteFolder(string $pathType)
    {
        $relativePath = $this->write->getRelativePath($this->directoryList->getPath($pathType));
        if ($this->write->isDirectory($relativePath)) {
            $this->write->delete($relativePath);
        }
    }
}

Class \Magento\Framework\App\ObjectManagerFactory (inject FileLock into GeneratedFiles):

(...)
    public function create(array $arguments)
    {
        $writeFactory = new \Magento\Framework\Filesystem\Directory\WriteFactory($this->driverPool);
        $lockManager = new \Magento\Framework\Lock\Backend\FileLock($this->driverPool->getDriver(DriverPool::FILE), BP);
        $generatedFiles = new GeneratedFiles($this->directoryList, $writeFactory, $lockManager);
        $generatedFiles->cleanGeneratedFiles();

(...)

This implementation is thought not to block any process execution, relying on only one of them to regenerate files, and to recover from possible errors by request a new regeneration for next process, unlock regenerate process and resuming execution.

[ihor-sviziev]

Hi @adrian-martinez-interactiv4,
Good idea 👍, I also was thinking about locking there

[ihor-sviziev]

Hi @lewisvoncken,
Could you update your PR with changes described by @adrian-martinez-interactiv4 ?

[adrian-martinez-interactiv4]

@ihor-sviziev let's wait a bit more for @lewisvoncken response, I can take this one if needed.

[ihor-sviziev]

I think in this case would be great to merge PR from you, as this is your solution :)

…

On Mon, 30 Sep 2019 at 16:57, adrian-martinez-interactiv4 < ***@***.***> wrote: @ihor-sviziev <https://github.com/ihor-sviziev> let's wait a bit more for @lewisvoncken <https://github.com/lewisvoncken> response, I can take this one if needed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24560?email_source=notifications&email_token=AAOJOUKIYOX35XJ6CZ5O673QMIAVPA5CNFSM4IVXDWA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD75XGCY#issuecomment-536572683>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOJOUJDE7KBM26LK5CKQZ3QMIAVPANCNFSM4IVXDWAQ> .

[adrian-martinez-interactiv4]

@ihor-sviziev PR in progress: #24892

[ihor-sviziev]

Hi @lewisvoncken,

Thank you for your contribution!
I'm closing this PR due to inactivity.

Also there was created new PR #24892 with requested changes, so we'll process it instead.

[m2-assistant]

Hi @lewisvoncken, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.