Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CPU at 100% with Mobilenet-gpu - 8 cores of i7-4770 and GeFORCE RTX 3060 #1283

Open
Jsalas424 opened this issue Jan 1, 2025 · 3 comments
Open

CPU at 100% with Mobilenet-gpu - 8 cores of i7-4770 and GeFORCE RTX 3060 #1283

Jsalas424 opened this issue Jan 1, 2025 · 3 comments

Comments

@Jsalas424
Copy link

Describe the bug

When I go to deploy the Mobilenet-gpu custom-build with docker-compose up -d, CPU usage goes to 100% and locks up the server.

To Reproduce

Deploy Mobilenet-gpu

Expected behavior

I expect GPU usage to increase instead of CPU

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Debian 11 LXC in Proxmox. This LXC concurrently runs Frigate with successful GPU access running TensorRT

Logs

Run those commands and attach result to the ticket:

docker ps

docker-compose logs

Had to cut out some log due to max character limits

root@frigate:/opt/compreface# docker-compose logs
Attaching to compreface-ui, compreface-admin, compreface-api, compreface-core, compreface-postgres-db
compreface-admin          | Listening for transport dt_socket at address: 5005
compreface-admin          | 
compreface-admin          |   .   ____          _            __ _ _
compreface-admin          |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
compreface-admin          | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
compreface-admin          |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
compreface-admin          |   '  |____| .__|_| |_|_| |_\__, | / / / /
compreface-admin          |  =========|_|==============|___/=/_/_/_/
compreface-admin          |  :: Spring Boot ::               (v2.5.13)
compreface-admin          | 
compreface-admin          | 2025-01-01 20:17:57.103  INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version      : HV000001: Hibernate Validator 6.2.3.Final
...
...
...
compreface-api            | 2025-01-01 20:18:20.623  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingsByIdUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.624  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_2
compreface-api            | 2025-01-01 20:18:20.627  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listEmbeddingsUsingGET_1
compreface-api            | 2025-01-01 20:18:20.637  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_2
compreface-api            | 2025-01-01 20:18:20.641  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeEmbeddingsUsingPOST_2
compreface-api            | 2025-01-01 20:18:20.645  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeFileUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.647  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: removeAllSubjectEmbeddingsUsingDELETE_1
compreface-api            | 2025-01-01 20:18:20.650  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: migrateUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.655  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: handleUsingGET_1
compreface-api            | 2025-01-01 20:18:20.668  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.672  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_3
compreface-api            | 2025-01-01 20:18:20.678  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeEmbeddingsUsingPOST_3
compreface-api            | 2025-01-01 20:18:20.681  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_3
compreface-api            | 2025-01-01 20:18:20.687  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: createSubjectUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.689  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectUsingDELETE_1
compreface-api            | 2025-01-01 20:18:20.691  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectsUsingDELETE_1
compreface-api            | 2025-01-01 20:18:20.693  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listSubjectsUsingGET_1
compreface-api            | 2025-01-01 20:18:20.696  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: renameSubjectUsingPUT_1
compreface-api            | 2025-01-01 20:18:20.713  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.718  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyBase64UsingPOST_1
compreface-api            | 2025-01-01 20:18:20.722  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyEmbeddingsUsingPOST_1
compreface-api            | 2025-01-01 20:18:20.772  INFO 7 --- [           main] com.exadel.frs.TrainServiceApplication   : Started TrainServiceApplication in 25.683 seconds (JVM running for 26.807)
compreface-api            | 2025-01-01 20:18:23.686  INFO 7 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
compreface-api            | 2025-01-01 20:18:23.686  INFO 7 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
compreface-api            | 2025-01-01 20:18:23.688  INFO 7 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
compreface-api            | Listening for transport dt_socket at address: 5005
compreface-api            | 
compreface-api            |   .   ____          _            __ _ _
compreface-api            |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
compreface-api            | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
compreface-api            |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
compreface-api            |   '  |____| .__|_| |_|_| |_\__, | / / / /
compreface-api            |  =========|_|==============|___/=/_/_/_/
compreface-api            |  :: Spring Boot ::               (v2.5.13)
compreface-api            | 
compreface-api            | 2025-01-01 20:23:28.975  INFO 7 --- [           main] com.exadel.frs.TrainServiceApplication   : Starting TrainServiceApplication v0.0.1-SNAPSHOT using Java 17.0.8 on 2c822965289e with PID 7 (/home/app.jar started by root in /)
compreface-api            | 2025-01-01 20:23:28.973  INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version      : HV000001: Hibernate Validator 6.2.3.Final
compreface-api            | 2025-01-01 20:23:28.981  INFO 7 --- [           main] com.exadel.frs.TrainServiceApplication   : The following 1 profile is active: "dev"
compreface-api            | 2025-01-01 20:23:29.134  WARN 7 --- [           main] o.s.b.c.config.ConfigDataEnvironment     : Property 'spring.profiles' imported from location 'class path resource [application.yml]' is invalid and should be replaced with 'spring.config.activate.on-profile' [origin: class path resource [application.yml] from app.jar - 97:13]
compreface-api            | 2025-01-01 20:23:32.525  INFO 7 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
compreface-api            | 2025-01-01 20:23:32.871  INFO 7 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 313 ms. Found 8 JPA repository interfaces.
compreface-api            | 2025-01-01 20:23:34.610  INFO 7 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=355f53b0-025e-31e7-98df-696283bbc190
compreface-api            | 2025-01-01 20:23:36.122  INFO 7 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'cacheConfig' of type [com.exadel.frs.core.trainservice.config.CacheConfig$$EnhancerBySpringCGLIB$$8a3fdb60] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
compreface-api            | 2025-01-01 20:23:37.566  INFO 7 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
compreface-api            | 2025-01-01 20:23:37.591  INFO 7 --- [           main] o.a.coyote.http11.Http11NioProtocol      : Initializing ProtocolHandler ["http-nio-8080"]
compreface-api            | 2025-01-01 20:23:37.592  INFO 7 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
compreface-api            | 2025-01-01 20:23:37.592  INFO 7 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.62]
compreface-api            | 2025-01-01 20:23:37.886  INFO 7 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
compreface-api            | 2025-01-01 20:23:37.886  INFO 7 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 8747 ms
compreface-api            | 2025-01-01 20:23:38.955  INFO 7 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
compreface-api            | 2025-01-01 20:23:39.376  INFO 7 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.4.33
compreface-api            | 2025-01-01 20:23:39.386  INFO 7 --- [           main] org.hibernate.cfg.Environment            : HHH000205: Loaded properties from resource hibernate.properties: {hibernate.bytecode.use_reflection_optimizer=false, hibernate.types.print.banner=false}
compreface-api            | 2025-01-01 20:23:40.142  INFO 7 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
compreface-api            | 2025-01-01 20:23:41.362  INFO 7 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
compreface-api            | 2025-01-01 20:23:41.629  INFO 7 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
compreface-api            | 2025-01-01 20:23:41.743  INFO 7 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect
compreface-api            | 2025-01-01 20:23:44.094  INFO 7 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
compreface-api            | 2025-01-01 20:23:44.124  INFO 7 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
compreface-api            | 2025-01-01 20:23:49.215  INFO 7 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
compreface-api            | 2025-01-01 20:23:49.456  INFO 7 --- [           main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)]
compreface-api            | 2025-01-01 20:23:50.246  INFO 7 --- [           main] o.a.coyote.http11.Http11NioProtocol      : Starting ProtocolHandler ["http-nio-8080"]
compreface-api            | 2025-01-01 20:23:50.262  INFO 7 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
compreface-api            | 2025-01-01 20:23:50.263  INFO 7 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed
compreface-api            | 2025-01-01 20:23:50.326  INFO 7 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Found 2 custom documentation plugin(s)
compreface-api            | 2025-01-01 20:23:50.501  INFO 7 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
compreface-api            | 2025-01-01 20:23:50.932  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_1
compreface-api            | 2025-01-01 20:23:50.939  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeEmbeddingsUsingPOST_1
compreface-api            | 2025-01-01 20:23:50.943  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_1
compreface-api            | 2025-01-01 20:23:51.043  INFO 7 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
compreface-api            | 2025-01-01 20:23:51.068  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getConfigUsingGET_1
compreface-api            | 2025-01-01 20:23:51.072  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getCheckDemoUsingGET_1
compreface-api            | 2025-01-01 20:23:51.089  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.094  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectBase64UsingPOST_1
compreface-api            | 2025-01-01 20:23:51.101  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.105  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingBase64UsingPOST_1
compreface-api            | 2025-01-01 20:23:51.108  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingByIdUsingDELETE_1
compreface-api            | 2025-01-01 20:23:51.112  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingsByIdUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.114  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_2
compreface-api            | 2025-01-01 20:23:51.121  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listEmbeddingsUsingGET_1
compreface-api            | 2025-01-01 20:23:51.138  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_2
compreface-api            | 2025-01-01 20:23:51.143  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeEmbeddingsUsingPOST_2
compreface-api            | 2025-01-01 20:23:51.147  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeFileUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.149  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: removeAllSubjectEmbeddingsUsingDELETE_1
compreface-api            | 2025-01-01 20:23:51.151  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: migrateUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.155  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: handleUsingGET_1
compreface-api            | 2025-01-01 20:23:51.163  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.165  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_3
compreface-api            | 2025-01-01 20:23:51.168  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeEmbeddingsUsingPOST_3
compreface-api            | 2025-01-01 20:23:51.170  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_3
compreface-api            | 2025-01-01 20:23:51.173  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: createSubjectUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.175  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectUsingDELETE_1
compreface-api            | 2025-01-01 20:23:51.176  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectsUsingDELETE_1
compreface-api            | 2025-01-01 20:23:51.177  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listSubjectsUsingGET_1
compreface-api            | 2025-01-01 20:23:51.179  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: renameSubjectUsingPUT_1
compreface-api            | 2025-01-01 20:23:51.188  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.191  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyBase64UsingPOST_1
compreface-api            | 2025-01-01 20:23:51.193  INFO 7 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyEmbeddingsUsingPOST_1
compreface-api            | 2025-01-01 20:23:51.222  INFO 7 --- [           main] com.exadel.frs.TrainServiceApplication   : Started TrainServiceApplication in 23.919 seconds (JVM running for 25.369)
compreface-postgres-db    | 2025-01-01 20:17:53.792 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
compreface-postgres-db    | 2025-01-01 20:17:53.792 UTC [1] LOG:  listening on IPv6 address "::", port 5432
compreface-postgres-db    | 2025-01-01 20:17:53.801 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
compreface-postgres-db    | 2025-01-01 20:17:53.825 UTC [25] LOG:  database system was shut down at 2025-01-01 20:17:45 UTC
compreface-postgres-db    | 2025-01-01 20:17:53.839 UTC [1] LOG:  database system is ready to accept connections
compreface-postgres-db    | 2025-01-01 20:23:26.036 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
compreface-postgres-db    | 2025-01-01 20:23:26.036 UTC [1] LOG:  listening on IPv6 address "::", port 5432
compreface-postgres-db    | 2025-01-01 20:23:26.054 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
compreface-postgres-db    | 2025-01-01 20:23:26.087 UTC [24] LOG:  database system was interrupted; last known up at 2025-01-01 20:17:53 UTC
compreface-postgres-db    | 2025-01-01 20:23:26.197 UTC [24] LOG:  database system was not properly shut down; automatic recovery in progress
compreface-postgres-db    | 2025-01-01 20:23:26.205 UTC [24] LOG:  redo starts at 0/1765388
compreface-postgres-db    | 2025-01-01 20:23:26.205 UTC [24] LOG:  invalid record length at 0/1765AF8: wanted 24, got 0
compreface-postgres-db    | 2025-01-01 20:23:26.205 UTC [24] LOG:  redo done at 0/1765AC0
compreface-postgres-db    | 2025-01-01 20:23:26.205 UTC [24] LOG:  last completed transaction was at log time 2025-01-01 20:18:36.066036+00
compreface-postgres-db    | 2025-01-01 20:23:26.261 UTC [1] LOG:  database system is ready to accept connections
compreface-postgres-db    | 2025-01-01 20:24:29.814 UTC [1] LOG:  received smart shutdown request
compreface-postgres-db    | 2025-01-01 20:24:29.819 UTC [1] LOG:  background worker "logical replication launcher" (PID 30) exited with exit code 1
compreface-postgres-db    | 2025-01-01 20:24:29.819 UTC [25] LOG:  shutting down
compreface-postgres-db    | 2025-01-01 20:24:29.852 UTC [1] LOG:  database system is shut down
compreface-core           | [uWSGI] getting INI configuration from uwsgi.ini
compreface-core           | *** Starting uWSGI 2.0.19 (64bit) on [Wed Jan  1 20:17:53 2025] ***
compreface-core           | compiled with version: 10.2.1 20210110 on 14 August 2023 20:24:09
compreface-core           | os: Linux-6.8.12-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-5 (2024-12-03T10:26Z)
compreface-core           | nodename: 128e5519ea8c
compreface-core           | machine: x86_64
compreface-core           | clock source: unix
compreface-core           | pcre jit disabled
compreface-core           | detected number of CPU cores: 8
compreface-core           | current working directory: /app/ml
compreface-core           | detected binary path: /usr/local/bin/uwsgi
compreface-core           | setgid() to 33
compreface-core           | setuid() to 33
compreface-core           | your processes number limit is 127764
compreface-core           | your memory page size is 4096 bytes
compreface-core           | detected max file descriptor number: 524288
compreface-core           | lock engine: pthread robust mutexes
compreface-core           | thunder lock: disabled (you can enable it with --thunder-lock)
compreface-core           | uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
compreface-core           | Python version: 3.8.17 (default, Jul 28 2023, 06:10:22)  [GCC 10.2.1 20210110]
compreface-core           | Python main interpreter initialized at 0x582d14b5d810
compreface-core           | python threads support enabled
compreface-core           | your server socket listen backlog is limited to 100 connections
compreface-core           | your mercy for graceful operations on workers is 60 seconds
compreface-core           | mapped 218712 bytes (213 KB) for 2 cores
compreface-core           | *** Operational MODE: preforking ***
compreface-core           | {"severity": "DEBUG", "message": "Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.", "request": null, "logger": "tensorflow", "module": "tpu_cluster_resolver", "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu_cluster_resolver.py\", line 32, in <module>\n    from cloud_tpu_client import client  # pylint: disable=g-import-not-at-top\nModuleNotFoundError: No module named 'cloud_tpu_client'\n", "build_version": "dev"}
compreface-core           | {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'facenet.Calculator', 'EXTRA_PLUGINS': ['facenet.LandmarksDetector', 'agegender.AgeDetector', 'agegender.GenderDetector', 'facenet.facemask.MaskDetector', 'facenet.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'facenet.FaceDetector', 'GPU_IDX': -1, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
compreface-core           | {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
compreface-core           | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x582d14b5d810 pid: 1 (default app)
compreface-core           | *** uWSGI is running in multiple interpreter mode ***
compreface-core           | spawned uWSGI master process (pid: 1)
compreface-core           | spawned uWSGI worker 1 (pid: 22, cores: 1)
compreface-core           | spawned uWSGI worker 2 (pid: 23, cores: 1)
compreface-core           | 2025-01-01 20:18:24.402557: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
compreface-core           | 2025-01-01 20:18:24.445248: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
compreface-core           | {"severity": "DEBUG", "message": "Found: BoundingBoxDTO(x_min=49, y_min=47, x_max=199, y_max=224, probability=0.9400066137313843, _np_landmarks=array([[104, 124],\n       [153, 124],\n       [131, 157],\n       [106, 177],\n       [146, 177]]))", "request": {"method": "GET", "path": "/status", "filename": "", "api_key": "", "remote_addr": "172.18.0.6"}, "logger": "src.services.facescan.plugins.facenet.facenet", "module": "facenet", "traceback": null, "build_version": "dev"}
compreface-core           | {"severity": "DEBUG", "message": "Found: BoundingBoxDTO(x_min=49, y_min=47, x_max=199, y_max=224, probability=0.9400066137313843, _np_landmarks=array([[104, 124],\n       [153, 124],\n       [131, 157],\n       [106, 177],\n       [146, 177]]))", "request": {"method": "GET", "path": "/status", "filename": "", "api_key": "", "remote_addr": "172.18.0.4"}, "logger": "src.services.facescan.plugins.facenet.facenet", "module": "facenet", "traceback": null, "build_version": "dev"}
compreface-core           | [uWSGI] getting INI configuration from uwsgi.ini
compreface-core           | *** Starting uWSGI 2.0.19 (64bit) on [Wed Jan  1 20:23:25 2025] ***
compreface-core           | compiled with version: 10.2.1 20210110 on 14 August 2023 20:24:09
compreface-core           | os: Linux-6.8.12-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-5 (2024-12-03T10:26Z)
compreface-core           | nodename: 128e5519ea8c
compreface-core           | machine: x86_64
compreface-core           | clock source: unix
compreface-core           | pcre jit disabled
compreface-core           | detected number of CPU cores: 8
compreface-core           | current working directory: /app/ml
compreface-core           | detected binary path: /usr/local/bin/uwsgi
compreface-core           | setgid() to 33
compreface-core           | setuid() to 33
compreface-core           | your processes number limit is 127764
compreface-core           | your memory page size is 4096 bytes
compreface-core           | detected max file descriptor number: 524288
compreface-core           | lock engine: pthread robust mutexes
compreface-core           | thunder lock: disabled (you can enable it with --thunder-lock)
compreface-core           | uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
compreface-core           | Python version: 3.8.17 (default, Jul 28 2023, 06:10:22)  [GCC 10.2.1 20210110]
compreface-core           | Python main interpreter initialized at 0x556a13781810
compreface-core           | python threads support enabled
compreface-core           | your server socket listen backlog is limited to 100 connections
compreface-core           | your mercy for graceful operations on workers is 60 seconds
compreface-core           | mapped 218712 bytes (213 KB) for 2 cores
compreface-core           | *** Operational MODE: preforking ***
compreface-core           | {"severity": "DEBUG", "message": "Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.", "request": null, "logger": "tensorflow", "module": "tpu_cluster_resolver", "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu_cluster_resolver.py\", line 32, in <module>\n    from cloud_tpu_client import client  # pylint: disable=g-import-not-at-top\nModuleNotFoundError: No module named 'cloud_tpu_client'\n", "build_version": "dev"}
compreface-core           | {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'facenet.Calculator', 'EXTRA_PLUGINS': ['facenet.LandmarksDetector', 'agegender.AgeDetector', 'agegender.GenderDetector', 'facenet.facemask.MaskDetector', 'facenet.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'facenet.FaceDetector', 'GPU_IDX': -1, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
compreface-core           | {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
compreface-core           | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x556a13781810 pid: 1 (default app)
compreface-core           | *** uWSGI is running in multiple interpreter mode ***
compreface-core           | spawned uWSGI master process (pid: 1)
compreface-core           | spawned uWSGI worker 1 (pid: 22, cores: 1)
compreface-core           | spawned uWSGI worker 2 (pid: 23, cores: 1)
compreface-core           | SIGINT/SIGQUIT received...killing workers...
compreface-core           | worker 1 buried after 1 seconds
compreface-core           | worker 2 buried after 1 seconds
compreface-core           | goodbye to uWSGI.
compreface-ui             | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
compreface-ui             | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
compreface-ui             | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
compreface-ui             | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
compreface-ui             | /docker-entrypoint.sh: Configuration complete; ready for start up
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET / HTTP/1.1" 200 1078 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /styles.7b70d6b4d311d52697eb.css HTTP/1.1" 200 12319 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /runtime-es2015.2f98b8256f9111c41ed5.js HTTP/1.1" 200 1397 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /polyfills-es2015.2da5f783f714f4c0074e.js HTTP/1.1" 200 12537 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /main-es2015.ee036b579cc361126068.js HTTP/1.1" 200 240507 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /assets/i18n/en.json?t=1735762702498 HTTP/1.1" 200 3977 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /common-es2015.1dc91cb5237cabf68a88.js HTTP/1.1" 200 4748 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /9-es2015.667f0258018000efe19b.js HTTP/1.1" 200 7209 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /assets/img/face-recognition-logo.svg HTTP/1.1" 200 7076 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /favicon.ico HTTP/1.1" 200 10313 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /admin/user/me HTTP/1.1" 401 113 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:22 +0000] "GET /12-es2015.04acbdc5287f2d1cf8c0.js HTTP/1.1" 200 3814 "http://frigate.trachenet:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:23 +0000] "GET /admin/status HTTP/1.1" 200 46 "http://frigate.trachenet:8000/login?redirect=%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | 192.168.1.204 - - [01/Jan/2025:20:18:23 +0000] "GET /assets/img/icons/check.svg HTTP/1.1" 200 172 "http://frigate.trachenet:8000/login?redirect=%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
compreface-ui             | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
compreface-ui             | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
compreface-ui             | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
compreface-ui             | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf
compreface-ui             | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
compreface-ui             | /docker-entrypoint.sh: Configuration complete; ready for start up

Additional context

I'm giving the LXC 8 core of i7-4770 CPU @ 3.40GHz and the GPU is a GeFORCE RTX 3060. Is this enough?

root@TracheNodeB:~# nvidia-smi
Wed Jan  1 15:31:03 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.142                Driver Version: 550.142        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:01:00.0 Off |                  N/A |
| 30%   45C    P2             23W /   70W |     286MiB /   6144MiB |      8%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2175541      C   frigate.detector.tensorrt                     170MiB |
|    0   N/A  N/A   2175706      C   ffmpeg                                        107MiB |
+-----------------------------------------------------------------------------------------+

Here's my config file, is there anything I can do to offload efforts to my GPU?

root@frigate:/opt/compreface# cat docker-compose.yml
version: '3.4'

volumes:
  postgres-data:

services:
  compreface-postgres-db:
    image: ${registry}compreface-postgres-db:${POSTGRES_VERSION}
    restart: always
    container_name: "compreface-postgres-db"
    environment:
      - POSTGRES_USER=${postgres_username}
      - POSTGRES_PASSWORD=${postgres_password}
      - POSTGRES_DB=${postgres_db}
    volumes:
      - postgres-data:/var/lib/postgresql/data

  compreface-admin:
    image: ${registry}compreface-admin:${ADMIN_VERSION}
    restart: always
    container_name: "compreface-admin"
    environment:
      - POSTGRES_USER=${postgres_username}
      - POSTGRES_PASSWORD=${postgres_password}
      - POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
      - SPRING_PROFILES_ACTIVE=dev
      - ENABLE_EMAIL_SERVER=${enable_email_server}
      - EMAIL_HOST=${email_host}
      - EMAIL_USERNAME=${email_username}
      - EMAIL_FROM=${email_from}
      - EMAIL_PASSWORD=${email_password}
      - ADMIN_JAVA_OPTS=${compreface_admin_java_options}
      - MAX_FILE_SIZE=${max_file_size}
      - MAX_REQUEST_SIZE=${max_request_size}B
    depends_on:
      - compreface-postgres-db
      - compreface-api

  compreface-api:
    image: ${registry}compreface-api:${API_VERSION}
    restart: always
    container_name: "compreface-api"
    depends_on:
      - compreface-postgres-db
    environment:
      - POSTGRES_USER=${postgres_username}
      - POSTGRES_PASSWORD=${postgres_password}
      - POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
      - SPRING_PROFILES_ACTIVE=dev
      - API_JAVA_OPTS=${compreface_api_java_options}
      - SAVE_IMAGES_TO_DB=${save_images_to_db}
      - MAX_FILE_SIZE=${max_file_size}
      - MAX_REQUEST_SIZE=${max_request_size}B
      - CONNECTION_TIMEOUT=${connection_timeout:-10000}
      - READ_TIMEOUT=${read_timeout:-60000}

  compreface-fe:
    image: ${registry}compreface-fe:${FE_VERSION}
    restart: always
    container_name: "compreface-ui"
    ports:
      - "8000:80"
    depends_on:
      - compreface-api
      - compreface-admin
    environment:
      - CLIENT_MAX_BODY_SIZE=${max_request_size}
      - PROXY_READ_TIMEOUT=${read_timeout:-60000}ms
      - PROXY_CONNECT_TIMEOUT=${connection_timeout:-10000}ms

  compreface-core:
    image: ${registry}compreface-core:${CORE_VERSION}
    restart: always
    container_name: "compreface-core"
    runtime: nvidia
    environment:
      - ML_PORT=3000
      - IMG_LENGTH_LIMIT=${max_detect_size}
      - UWSGI_PROCESSES=${uwsgi_processes:-1}
      - UWSGI_THREADS=${uwsgi_threads:-1}

Add any other context about the problem here.
@Jsalas424
Copy link
Author

Jsalas424 commented Jan 1, 2025 •
edited
Loading

Performance plots when I start the container

image

@Jsalas424
Copy link
Author

Jsalas424 commented Jan 1, 2025 •
edited
Loading

It does look like CompreFace is using my GPU but it's still locking up my CPU. This is using the docker-compose.yml from the Master Branch

root@Server:~# nvidia-smi
Wed Jan  1 15:58:55 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.142                Driver Version: 550.142        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:01:00.0 Off |                  N/A |
| 30%   39C    P8             10W /   70W |     366MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2191813      C   frigate.detector.tensorrt                     170MiB |
|    0   N/A  N/A   2191907      C   ffmpeg                                        107MiB |
|    0   N/A  N/A   2194835      C   uwsgi                                          78MiB |
+-----------------------------------------------------------------------------------------+

Is my CPU just inadequate?

@emersonsc
Copy link

It does look like CompreFace is using my GPU but it's still locking up my CPU. This is using the docker-compose.yml from the Master Branch

root@Server:~# nvidia-smi
Wed Jan  1 15:58:55 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.142                Driver Version: 550.142        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:01:00.0 Off |                  N/A |
| 30%   39C    P8             10W /   70W |     366MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2191813      C   frigate.detector.tensorrt                     170MiB |
|    0   N/A  N/A   2191907      C   ffmpeg                                        107MiB |
|    0   N/A  N/A   2194835      C   uwsgi                                          78MiB |
+-----------------------------------------------------------------------------------------+

Is my CPU just inadequate?

No, I have a 12900k in my homelab server and I noticed for some reason all the sudden compreface is putting my CPU on its knees and maxing every core even though its also putting my GPU to work. SOMETHING broke compreface, but since this project seems to be abandoned, its likely not going to get fixed. Its a shame, this was a cool bit of software too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jsalas424 @emersonsc