Merge pull request #76 from vtsykun/feature/perf-for-composer2

Performance improvement when the number of packets is large

README.md

@@ -47,6 +47,8 @@ Table of content
         - [JIRA issue fix version](/docs/webhook.md#jira-create-a-new-release-and-set-fix-version)
         - [Gitlab setup auto webhook](/docs/webhook.md#gitlab-auto-webhook)
 - [Ssh key access](#ssh-key-access-and-composer-oauth-token)
+- [Configuration](#configuration)
+- [LPAD Authenticating](/docs/authentication-ldap.md)
 - [Update Webhooks](#update-webhooks)
     - [Github](#github-webhooks)
     - [GitLab](#gitlab-service)
@@ -56,7 +58,7 @@ Table of content
     - [Custom webhook format](#custom-webhook-format-transformer)
 - [Mirroring Composer repos](docs/usage/mirroring.md)
 - [Usage](#usage-and-authentication)
-    - [Create admin user](#create-admin-user)
+    - [Create admin user](#create-admin-and-maintainer-users)
 
 Demo
 ----
@@ -124,18 +126,23 @@ Installation
 1. Clone the repository
 2. Install dependencies: `composer install`
 3. Create .env.local and copy needed environment variables into it, see docker Environment variables section
-4. Run `bin/console doctrine:schema:create` to setup the DB
+4. Run `bin/console doctrine:schema:update --force --complete` to set up the DB
 5. Create admin user via console.
 
 ```
 php bin/console packagist:user:manager username [email protected] --password=123456 --admin 
 ```
 
 6. Enable cron tabs and background jobs.
-Enable crontab `crontab -e -u www-data` 
+Enable crontab `crontab -e -u www-data` or use Docker friendly build-in cron demand runner.
 
 ```
-* * * * * /var/www/packagist/bin/console --env=prod okvpn:cron >> /dev/null
+* * * * * /var/www/packagist/bin/console okvpn:cron >> /dev/null
+```
+
+Example, run cron as background process without crontab. Can use with supervisor.
+```
+bin/console okvpn:cron --demand
 ```
 
 Setup Supervisor to run worker.
@@ -246,6 +253,72 @@ We disable usage GitHub API by default to force use ssh key or clone the reposit
 it would with any other git repository. You can enable it again with env option `GITHUB_NO_API` 
 [see here](https://getcomposer.org/doc/06-config.md#use-github-api).
 
+Configuration
+-------------
+
+In order to add a configuration add a file with any name to the folder `config/packages/*`. 
+The config will merge with default values in priority sorted by filename.
+
+The configuration for Docker installation is available at `/data/config.yaml`.
+Also, you can use docker volume to add config directly at path `config/packages/ldap.yaml`.
+
+```yaml
+...
+        volumes:
+            - .docker:/data
+            - ${PWD}/ldap.yaml:/var/www/packagist/config/packages/ldap.yaml
+```
+
+Where `/var/www/packagist/` default ROOT for docker installation.
+
+Full example of configuration.
+
+```yaml
+packeton:
+    github_no_api: '%env(bool:GITHUB_NO_API)%' # default true
+    rss_max_items: 30
+    archive: true
+    
+    # default false
+    anonymous_access: '%env(bool:PUBLIC_ACCESS)%'
+
+    anonymous_archive_access: '%env(bool:PUBLIC_ACCESS)%' # default false
+    
+    archive_options:
+        format: zip
+        basedir: '%env(resolve:PACKAGIST_DIST_PATH)%'
+        endpoint: '%env(PACKAGIST_DIST_HOST)%' # default auto detect by host headers 
+        include_archive_checksum: false
+    
+    # disable by default 
+    jwt_authentication:
+        algo: EdDSA
+        private_key: '%kernel.project_dir%/var/jwt/eddsa-key.pem'
+        public_key: '%kernel.project_dir%/var/jwt/eddsa-public.pem'
+        passphrase: ~
+
+    # See mirrors section
+    mirrors: ~
+    
+    metadata:
+        format: auto # Default, see about metadata.
+        info_cmd_message: ~ # Bash logo, example - \u001b[37;44m#StandWith\u001b[30;43mUkraine\u001b[0m
+```
+
+### Metadata format.
+
+Packeton support metadata for Composer 1 and 2. For performance reasons, for Composer 1 uses metadata
+depending on the user-agent header: `providers-lazy-url` if ua != 1; `provider-includes` if ua == 1;
+
+| Format strategy | UA 1                           | UA 2                            | UA is NULL                      |
+|-----------------|--------------------------------|---------------------------------|---------------------------------|
+| auto            | provider-includes metadata-url | providers-lazy-url metadata-url | providers-lazy-url metadata-url |
+| only_v1         | provider-includes              | provider-includes               | provider-includes               |
+| only_v2         | metadata-url                   | metadata-url                    | metadata-url                    |
+| full            | provider-includes metadata-url | provider-includes metadata-url  | provider-includes metadata-url  |
+
+Where `UA 1` - Composer User-Agent = 1. `UA 2` - Composer User-Agent = 2.
+
 Update Webhooks
 ---------------
 You can use GitLab, Gitea, GitHub, and Bitbucket project post-receive hook to keep your packages up to date 
@@ -415,10 +488,10 @@ Configure this private repository in your `composer.json`.
 
 **Application Roles**
 
-- ROLE_USER - minimal access level, these users only can read metadata only for selected packages.
-- ROLE_FULL_CUSTOMER - Can read all packages metadata.
-- ROLE_MAINTAINER -  Can submit a new package and read all metadata.
-- ROLE_ADMIN - Can create a new customer users, management webhooks and credentials.
+- `ROLE_USER` - minimal access level, these users only can read metadata only for selected packages.
+- `ROLE_FULL_CUSTOMER` - Can read all packages metadata.
+- `ROLE_MAINTAINER` -  Can submit a new package and read all metadata.
+- `ROLE_ADMIN` - Can create a new customer users, management webhooks and credentials.
 
 You can create a user and then promote to admin or maintainer via console using fos user bundle commands.
 

config/services.yaml

@@ -104,6 +104,10 @@ services:
         arguments:
             $config: '%packeton_archive_opts%'
 
+    Packeton\Package\InMemoryDumper:
+        arguments:
+            $config: '%packeton_dumper_opts%'
+
     Packeton\DBAL\OpensslCrypter:
         public: true
         arguments:

src/Composer/Cache/MetadataCache.php

@@ -16,7 +16,7 @@ public function __construct(
     ) {
     }
 
-    public function get(string $key, callable $callback, int $lastModify = null)
+    public function get(string $key, callable $callback, int $lastModify = null, callable $needClearCache = null)
     {
         // Use host key to prevent Cache Poisoning attack, if dist URL generated dynamic.
         // But for will protection must be used trusted_hosts
@@ -27,9 +27,12 @@ public function get(string $key, callable $callback, int $lastModify = null)
         @[$ctime, $data] = $item->get();
 
         $needRefresh = false;
-        if ($lastModify !== null) {
+        if (null !== $lastModify) {
             $needRefresh = $ctime < $lastModify || $ctime + $this->maxTtl < time();
         }
+        if (null !== $needClearCache) {
+            $needRefresh = $needRefresh || $needClearCache($data);
+        }
 
         if (!$item->isHit() || $needRefresh || empty($data)) {
             $data = $callback($item);

src/Composer/MetadataFormat.php

@@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Packeton\Composer;
+
+enum MetadataFormat: string
+{
+    case AUTO = 'auto';
+    case ONLY_V1 = 'only_v1';
+    case ONLY_V2 = 'only_v2';
+    case FULL = 'full';
+
+    public function providerIncludes(int $version = null): bool
+    {
+        return match(true) {
+            $this === MetadataFormat::AUTO && $version === 1, $this === MetadataFormat::ONLY_V1, $this === MetadataFormat::FULL => true,
+            default => false
+        };
+    }
+
+    public function lazyProviders(int $version = null): bool
+    {
+        return match(true) {
+            $this === MetadataFormat::AUTO && $version !== 1 => true,
+            default => false
+        };
+    }
+
+    public function metadataUrl(int $version = null): bool
+    {
+        return match(true) {
+            $this === MetadataFormat::ONLY_V1 => false,
+            default => true
+        };
+    }
+}

src/Controller/PackageController.php

@@ -501,6 +501,7 @@ public function deletePackageVersionAction(Request $req, $versionId): Response
             throw new AccessDeniedException;
         }
 
+        $this->providerManager->setLastModify($package->getName());
         $repo->remove($version);
         $this->registry->getManager()->flush();
         $this->registry->getManager()->clear();

src/Controller/ProviderController.php

@@ -10,7 +10,9 @@
 use Packeton\Entity\Package;
 use Packeton\Entity\Version;
 use Packeton\Model\PackageManager;
+use Packeton\Model\ProviderManager;
 use Packeton\Service\DistManager;
+use Packeton\Util\UserAgentParser;
 use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
 use Symfony\Component\HttpFoundation\BinaryFileResponse;
 use Symfony\Component\HttpFoundation\Request;
@@ -24,21 +26,27 @@ class ProviderController extends AbstractController
 
     public function __construct(
         private readonly PackageManager $packageManager,
+        private readonly ProviderManager $providerManager,
         private readonly ManagerRegistry $registry,
-    ){
+    ) {
     }
 
     #[Route('/packages.json', name: 'root_packages', defaults: ['_format' => 'json'], methods: ['GET'])]
     public function packagesAction(Request $request): Response
     {
-        $rootPackages = $this->packageManager->getRootPackagesJson($this->getUser());
+        $response = new JsonResponse([]);
+        $response->setLastModified($this->providerManager->getRootLastModify());
+        if ($response->isNotModified($request)) {
+            return $response;
+        }
+
+        $ua = new UserAgentParser($request->headers->get('User-Agent'));
+        $apiVersion = $request->query->get('ua') ? (int) $request->query->get('ua') : $ua->getComposerMajorVersion();
 
-        $response = new JsonResponse($rootPackages);
+        $rootPackages = $this->packageManager->getRootPackagesJson($this->getUser(), $apiVersion);
+
+        $response->setData($rootPackages);
         $response->setEncodingOptions(JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT);
-        if ($lastModify = $this->packageManager->getLastModify()) {
-            $response->setLastModified($lastModify);
-            $response->isNotModified($request);
-        }
 
         return $response;
     }
@@ -120,20 +128,22 @@ public function packageAction(string $package): Response
     )]
     public function packageV2Action(Request $request, string $package): Response
     {
+        $response = new JsonResponse([]);
+        $response->setLastModified($this->providerManager->getLastModify($package));
+        if ($response->isNotModified($request)) {
+            return $response;
+        }
+
         $isDev = str_ends_with($package, '~dev');
         $package = preg_replace('/~dev$/', '', $package);
 
-        $package = $this->packageManager->getPackageV2Json($this->getUser(), $package, $isDev, $lastModified);
+        $package = $this->packageManager->getPackageV2Json($this->getUser(), $package, $isDev);
         if (!$package) {
             return $this->createNotFound();
         }
 
-        $response = new JsonResponse($package);
         $response->setEncodingOptions(\JSON_UNESCAPED_SLASHES);
-        if ($lastModified !== null) {
-            $response->setLastModified(new \DateTime($lastModified));
-            $response->isNotModified($request);
-        }
+        $response->setData($package);
 
         return $response;
     }

src/Controller/ProxiesController.php

@@ -18,6 +18,7 @@
 use Packeton\Mirror\Utils\MirrorTextareaParser;
 use Packeton\Mirror\Utils\MirrorUIFormatter;
 use Packeton\Model\PackageManager;
+use Packeton\Model\ProviderManager;
 use Packeton\Service\JobScheduler;
 use Packeton\Util\HtmlJsonHuman;
 use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
@@ -38,7 +39,7 @@ public function __construct(
         private readonly JobScheduler $jobScheduler,
         private readonly MirrorPackagesValidate $mirrorValidate,
         private readonly MetadataMinifier $metadataMinifier,
-        private readonly PackageManager $packageManager,
+        private readonly ProviderManager $providerManager,
     ) {
     }
 
@@ -204,7 +205,7 @@ protected function getProxyData(RemoteProxyRepository $repo): array
         $repoUrl = $this->generateUrl('mirror_index', ['alias' => $config->getAlias()], UrlGeneratorInterface::ABSOLUTE_URL);
 
         $rpm = $repo->getPackageManager();
-        $privatePackages = $this->packageManager->getPackageNames();
+        $privatePackages = $this->providerManager->getPackageNames();
 
         $packages = MirrorUIFormatter::getGridPackagesData($rpm->getApproved(), $rpm->getEnabled(), $privatePackages);
 

src/DependencyInjection/Configuration.php

@@ -3,6 +3,7 @@
 namespace Packeton\DependencyInjection;
 
 use Firebase\JWT\JWT;
+use Packeton\Composer\MetadataFormat;
 use Symfony\Component\Config\Definition\Builder\ArrayNodeDefinition;
 use Symfony\Component\Config\Definition\Builder\NodeDefinition;
 use Symfony\Component\Config\Definition\Builder\TreeBuilder;
@@ -27,6 +28,12 @@ public function getConfigTreeBuilder()
             ->children()
                 ->booleanNode('github_no_api')->end()
                 ->scalarNode('rss_max_items')->defaultValue(40)->end()
+                ->arrayNode('metadata')
+                    ->children()
+                        ->enumNode('format')->values(array_map(fn($o) => $o->value, MetadataFormat::cases()))->end()
+                        ->scalarNode('info_cmd_message')->end()
+                    ->end()
+                ->end()
                 ->booleanNode('anonymous_access')->defaultFalse()->end()
                 ->booleanNode('anonymous_archive_access')->defaultFalse()->end()
                 ->booleanNode('archive')

src/DependencyInjection/PacketonExtension.php

@@ -5,6 +5,7 @@
 namespace Packeton\DependencyInjection;
 
 use Packeton\Attribute\AsWorker;
+use Packeton\Package\InMemoryDumper;
 use Symfony\Component\DependencyInjection\ChildDefinition;
 use Symfony\Component\DependencyInjection\ContainerBuilder;
 use Symfony\Component\DependencyInjection\Exception\LogicException;
@@ -25,6 +26,8 @@ public function load(array $configs, ContainerBuilder $container)
             $container->setParameter('packeton_archive_opts', $config['archive_options'] ?? []);
         }
 
+        $container->setParameter('packeton_dumper_opts', $config['metadata'] ?? []);
+
         $hasPublicMirror = array_filter($config['mirrors'] ?? [] , fn ($i) => $i['public_access'] ?? false);
         $container->setParameter('anonymous_mirror_access', (bool) $hasPublicMirror);
 

src/EventListener/DoctrineListener.php

@@ -12,7 +12,7 @@
 use Packeton\Entity\Package;
 use Packeton\Entity\User;
 use Packeton\Entity\Version;
-use Packeton\Model\PackageManager;
+use Packeton\Model\ProviderManager;
 use Packeton\Service\DistConfig;
 use Symfony\Component\HttpFoundation\RequestStack;
 
@@ -28,7 +28,7 @@ class DoctrineListener
 
     public function __construct(
         private readonly RequestStack $requestStack,
-        private readonly PackageManager $packageManager,
+        private readonly ProviderManager $providerManager,
     ){
     }
 
@@ -65,7 +65,7 @@ public function onFlush(OnFlushEventArgs $args): void
         foreach ($changes as $object) {
             $class = ClassUtils::getClass($object);
             if (isset(self::$trackLastModifyClasses[$class])) {
-                $this->packageManager->setLastModify();
+                $this->providerManager->setRootLastModify();
                 return;
             }
         }

src/Mirror/RemoteProxyRepository.php

@@ -341,6 +341,7 @@ public function packageKey(string $package, string $hash = null): string
 
         $vendor = $this->safeName($vendor);
         $pkg = $this->safeName($pkg) ?: '_null_';
+        $hash = $hash ? $this->safeName($hash) : null;
 
         return $vendor . $this->ds . $pkg . ($hash ? self::HASH_SEPARATOR . $hash : '') . '.json.gz';
     }