Incomplete folders restored when using Composer 2

[pfrenssen]

When I am installing Drupal 7 alongside a number of modules from a freshly cloned repo it appears that the modules folder seems to get backed up in a partially completed state, and then later on restored. This overwrites part of the modules. The modules folder ends up with a bunch of empty folders and missing modules.

The console output indicates that some folders are being overwritten:

[...]
  - Installing drush/drush (8.4.8): Extracting archive
  - Installing symfony/filesystem (v3.4.47): Extracting archive
Files of installed package were overwritten with preserved path web/sites/all/modules/contrib!
Files of installed package were overwritten with preserved path web/sites/all/modules/contrib!

This happens only with Composer 2, so possibly this is caused by the parallel installation of the packages. Possibly some modules get installed before Drupal core itself is installed? I tried setting the COMPOSER_MAX_PARALLEL_HTTP=1 environment variable but this doesn't help. It only throttles the downloading of the packages, not the installation.

I am not using composer-patches. It happens with the latest version of Composer (2.0.12).

Here are the relevant parts of the composer.json file that is causing the problem:

{
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/7"
        }
    ],
    "require": {
        "php": "^7.4",
        "ext-curl": "*",
        "ext-gd": "*",
        "ext-json": "*",
        "ext-openssl": "*",
        "ext-pdo": "*",
        "ext-xml": "*",
        "composer/installers": "^1.2",
        "composer/semver": "^1.4",
        "drupal-composer/preserve-paths": "^0.1",
        "drupal/admin_menu": "3.0-rc6",
        "drupal/bootstrap": "^3.26",
        "drupal/composer_autoloader": "^1.0",
        "drupal/ctools": "1.15",
        "drupal/drupal": "^7.80",
        "drupal/ds": "2.16",
        "drupal/email": "1.3",
        "drupal/entity": "1.9",
        "drupal/entity_token": "1.9",
        "drupal/features": "2.11",
        "drupal/field_collection": "1.0-beta13",
        "drupal/field_group": "1.6",
        "drupal/googleanalytics": "2.6",
        "drupal/isotope": "2.0",
        "drupal/jquery_update": "2.7",
        "drupal/l10n_update": "1.2",
        "drupal/libraries": "2.5",
        "drupal/link": "1.7",
        "drupal/menu_block": "2.8",
        "drupal/module_filter": "2.2",
        "drupal/nodeblock": "1.7",
        "drupal/pathauto": "1.0-rc1",
        "drupal/strongarm": "2.0",
        "drupal/token": "1.7",
        "drupal/transliteration": "3.2",
        "drupal/video_embed_field": "2.0-beta11",
        "drupal/views": "3.22",
        "drupal/webform": "4.22",
        "drupal/weight": "3.1",
        "drupal/wysiwyg": "2.0",
        "drupal/xmlsitemap": "2.6",
        "drupal/youtube": "1.7",
        "drush/drush": "^8.0",
        "symfony/filesystem": "~2.7|^3",
        "webflo/drupal-finder": "^1.0.0"
    },
    "conflict": {
        "drupal/core": "8.*"
    },
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-create-project-cmd": [
            "DrupalProject\\composer\\ScriptHandler::removeInternalFiles"
        ]
    },
    "extra": {
        "installer-paths": {
            "web/": ["type:drupal-core"],
            "web/profiles/{$name}/": ["type:drupal-profile"],
            "web/sites/all/drush/{$name}/": ["type:drupal-drush"],
            "web/sites/all/libraries/{$name}/": ["type:drupal-library"],
            "web/sites/all/modules/contrib/{$name}/": ["type:drupal-module"],
            "web/sites/all/themes/{$name}/": ["type:drupal-theme"]
        },
        "preserve-paths": [
            "web/sites/all/modules/contrib",
            "web/sites/all/modules/features",
            "web/sites/all/themes/contrib",
            "web/sites/default/drushrc.php",
            "web/sites/default/files",
            "web/sites/default/settings.php"
        ]
    }
}

[generalredneck]

quick "me too" on this one... Still a problem on 2.1.3

[generalredneck]

From what I can tell... composer is running the pre-package-install hook prior to most things being installed. All the pre-package install (and install) all happen first (and serially). Then all the Post package install hooks are run together serially.

In my case since the paths were preserved as they were near the beginning of the install the packages that were installed after drupal/drupal was do not exist in the preserved cache folder. Then when the post package-install hooks happen later, all the installed packages are wiped out by the folder that in preserve.

This is different from Composer 1 because the each package goes through the process of pre-install to post-install per package. Therefore you get the state of the paths as they were directly before the installation and then put back immediately after before the next item is installed.

I really don't know how to solve this problem with that being the case.

[generalredneck]

as you can see I've added composer/composer#9983. I really don't know what to do until then.

[generalredneck]

So in the following comment: composer/composer#9983 (comment)

There is no intention of supporting non-concurrency and the way that the events fire seems intentional. I'm working on work arounds currently... but I think the continued viability of this plugin may need to be looked at. Since Drupal 7 is coming close to EOL and the amount of composer installs are likely even a smaller portion of the D7 sites that are still around... We may consider duct taping a solution.

That is to say if not other projects use this plugin for their frameworks that we don't know about.

I've got something in the works and will post what I did to get around this for others

[generalredneck]

So here's what I ended up doing... Note that YMMV and you will need to configure a few things in the script. I didn't go through the effort of seeing if I could get the composer extra configuration through even being passed in to each of the scripts... but here's what I did. This assumes that I'm moving everything that used to be in web/sites/all to temp/sites/all in my composer file.

EDIT: Removed this code sample and will post the new one in a new comment cause this turned out to not work well in autoloading.

[generalredneck]

I wanted to come back and post my final solution as my previous comment didn't work well for autoloading of libraries. There was a typo in my search replace. However, I did something that simplified things a bit.

First...

Change only drupal core in your installer paths

  "installer-paths": {
            "temp/core": ["type:drupal-core"],

Add The following to scripts/composer/ScriptHandler.php

  public static function moveInCore() {
    $root = getcwd();
    $source_dir = $root . '/temp/core';
    if (!file_exists($source_dir)) {
      return;
    }
    // CHANGE THIS AS YOU NEED (E.G. docroot).
    $dest_dir = $root . '/web';  
    if (!is_dir($dest_dir)) {
      mkdir($dest_dir, 0755, TRUE);
    }
    // ADD ANY SCAFFOLDING YOU WANT TO KEEP IN THE REPO.
    $excluded_files = [
      'sites',
      '.htaccess',
      'robots.txt',
      'profiles',
    ];
    $files = array_diff(scandir($source_dir), array('.', '..'));
    foreach ($files as $file) {
      $old_file = $source_dir . '/' . $file;
      $new_file = $dest_dir . '/' . $file;
      if (in_array($file, $excluded_files)) {
        continue;
      }
      if (file_exists($new_file)) {
        (is_dir($new_file) && !is_link($new_file)) ? self::delTree($new_file) : unlink($new_file);
      }
      rename($old_file, $new_file);
    }
    self::delTree($root . '/temp');
  }

  public static function delTree($dir) {
    $files = array_diff(scandir($dir), array('.', '..'));
    foreach ($files as $file) {
      (is_dir("$dir/$file") && !is_link("$dir/$file")) ? $result = self::delTree("$dir/$file") : $result = unlink("$dir/$file");
    }
    return rmdir($dir);
  }

Then add

            "DrupalProject\\composer\\ScriptHandler::moveInCore",
            "find vendor/composer -type f | xargs sed -i  's/\\/temp\\/core/\\/web/g'"

to both your post-install-cmd and post-update-cmd, changing web in the find command to the directory you need (e.g. docroot).

Hope this helps.

[mfb]

I guess this is also happening with DrupalCI on drupal.org? I noticed that tests are failing for one of my D7 contrib modules on PHP 7.3 thru PHP 8.1 (which use Composer 2) while passing on PHP 7.2 (which uses Composer 1). It appears that dependencies aren't available - presumably empty module directory - and tests fail during the setUp method.. https://www.drupal.org/pift-ci-job/2420090

[olstjos]

@mfb thanks for hunting this down.

[mfb]

Well, the problem I've seen with DrupalCI and D7 contrib modules might be unrelated to this issue - I posted a potential fix at https://www.drupal.org/project/drupalci_testbot/issues/3294386

[generalredneck]

@mfb right... the error you posted was related to a totally different deadline due to a security fix starting with composer 2.2.0. You can see related issues like https://www.drupal.org/project/drupal/issues/3255749 and the original documentation at https://getcomposer.org/doc/06-config.md#allow-plugins and a blog post announcing the feature at https://blog.packagist.com/composer-2-2/

[mfb]

yes I'm very familiar from local development, just took me a while to track the drupalci issue down to it :)

[tormi]

@generalredneck & others, there's the new kid on the block that solves the problem via symlinks: https://github.com/druidfi/mona-plugin