Python error on startup

[eflaten]

What is the bug or the crash?

There is Python Error displayes during startup. Here it is:

An error occurred during execution of following code:
is_startup_script_executed = run_startup_script(pyqgstart)

Traceback (most recent call last):
File "", line 1, in
File "QgsPythonUtilsImpl::checkSystemImports [run_startup_script]", line 7, in run_startup_script
ModuleNotFoundError: No module named 'qgis'

Python version:
3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 19:00:53) [Clang 16.0.6 ]

QGIS version:
3.38.0-Grenoble 'Grenoble', exported

Python path:
['/Applications/QGIS-3.38.app/Contents/lib/python311.zip', '/Applications/QGIS-3.38.app/Contents/lib/python3.11', '/Applications/QGIS-3.38.app/Contents/lib/python3.11/lib-dynload', '/Applications/QGIS-3.38.app/Contents/lib/python3.11/site-packages']

Steps to reproduce the issue

Doubleclick the qgis icon and wait

Versions

QGIS version
3.38.0-Grenoble
QGIS code branch
Release 3.38
Qt version
5.15.8
Python version
3.11.9
Compiled against GDAL/OGR
3.9.0
Running against GDAL/OGR
3.9.1
PROJ version
9.4.1
EPSG Registry database version
v11.006 (2024-03-13)
GEOS version
3.12.1-CAPI-1.18.1
SQLite version
3.46.0
Compiled against PDAL
2.7.1
Running against PDAL
2.7.2
PostgreSQL client version
unknown
SpatiaLite version
5.1.0
QWT version
6.3.0
QScintilla2 version
2.14.1
OS version
macOS 10.16

Active Python plugins
beePen
0.2.5
batch_gps_importer
version 1.0.1
wfsclient
0.9.11
qgis2web
3.21.0
processing
2.12.99
grassprovider
2.12.99
db_manager
0.1.20
MetaSearch
0.3.6

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

Its the new macos installer:
https://github.com/opengisch/qgis-conda-builder/releases

No response

[agiudiceandrea]

@eflaten, thanks for reporting. Does the issue occur also using a new QGIS user profile?
I think you need to report such issue to the qgis-conda-builder developers at https://github.com/opengisch/qgis-conda-builder.
Perhaps @m-kuhn may be of help here.

[eflaten]

Yes. Same error when starting up with a new profile.
Sidenote. There was no way to enter a new user profile in the settings/user profiles menu. So I just made an folder manually and qgis used that folder as a profile and filles it with default stuff.

[github-actions]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale".
If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue.
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue.
If there is no further activity on this issue, it will be closed in a week.

[github-actions]

While we hate to see this happen, this issue has been automatically closed because it has not had any activity in the last 42 days despite being marked as feedback. If this issue should be reconsidered, please follow the guidelines in the previous comment and reopen this issue.
Or, if you have any further questions, there are also further support channels that can help you.

[thbaumann]

Same issue here with Windows as OS (QGIS 3.38, installed with the OSGEO Network installer). Seems like a timing problem, that the module qgis is not yet available when the run_startup_script(pyqgstart) is executed.

If QGIS is started, I can also import qgis in the console without getting this error.

[agiudiceandrea]

@thbaumann, thanks for reporting. May you please indicated the exact QGIS version installed?

[thbaumann]

@agiudiceandrea : I have

QGIS-Version:
3.38.2-Grenoble 'Grenoble', 130c432

and

QGIS-Version:
3.38.3-Grenoble 'Grenoble', 37f9e6e

The Problem exists in both versions.

[ostr00000]

I suspect that there is used a different search path for python for module qgis. This path is probably set/changed in a provided script.
Unfortunately, I am not able to test on Windows. @thbaumann Can we find out what path is used as PYQGIS_STARTUP, and what is the content of that file?

I was hesitant about logging using qgisQgsMessageLog at that point. #56827 (comment)

I think about a few possible fixes:
a) deffer logs that use QgsMessageLog until startup script is executed,
b) log files using print (I know that there may be another errors on window regarding stdout - #57173),
c) use deferred QgsMessageLog logs, but in case of error (missing file provided in PYQGIS_STARTUP), try to use qgis module.

[thbaumann]

@ostr00000 : The path of PYQGIS_STARTUP is a file on a networkdrive mapped as drive letter A:

A:\qgis\helper\pyqgis_startup\pre_startup.py

The content of this script is a md5 check and a shutil copy to distribute the current startup.py to all users:

import sys, os, subprocess, shutil, hashlib

appdataverzeichnis=os.getenv('APPDATA')

qgis3_folder='{0}/QGIS/QGIS3/'.format(appdataverzeichnis)

source_file='A:/qgis/helper/startup_py/QGIS3/startup.py'
dest_file='{0}/startup.py'.format(qgis3_folder)
backup_file='{0}/startup_overwritten.py'.format(qgis3_folder)

def copy_startup():
    if os.path.isfile(dest_file):
        shutil.copy2(dest_file, backup_file)
    os.makedirs(os.path.dirname(qgis3_folder), exist_ok=True)
    shutil.copy2(source_file, qgis3_folder)
    
if sys.version[0] == '3':
    if os.path.isfile(dest_file):
        digests = []
        for filename in [source_file, dest_file]:
            hasher = hashlib.md5()
            with open(filename, 'r') as f:
                buf = f.read()
                hasher.update(buf.encode('utf-8'))
                a = hasher.hexdigest()
                digests.append(a)
                #print(a)

        if(digests[0] != digests[1]):
            copy_startup()
    else:
        copy_startup()
elif sys.version[0] == '2':
    pass

[ostr00000]

The above startup script has no sys.path manipulation, so it is irrelevant (BTW nice synchronization script).

---

I was able to reproduce this error, but only on QGIS version installed by OSGeo4W network installer (on my dev installation there is no such error).
The error occurs only when PYQGIS_STARTUP is set, otherwise we quit early this function.

---

I investigated sys.path differences between:

path listed on error dialog

  [
'C:\\OSGeo4W\\apps\\grass\\grass83\\etc\\python', 
'C:\\OSGeo4W\\apps\\gdal-dev\\lib\\site-packages',
'C:\\OSGeo4W\\bin', 
'C:\\OSGeo4W\\bin\\python312.zip', 
'C:\\OSGeo4W\\apps\\Python312\\DLLs', 
'C:\\OSGeo4W\\apps\\Python312\\Lib', 
'C:\\OSGeo4W\\apps\\Python312', 
'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages', 
'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\win32',
'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\win32\\lib', 
'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\Pythonwin'
]

path listed using QGIS python console (`pprint(sys.path)`)

[
# C++ function `pythonPath`
'C:/OSGeo4W/apps/qgis-dev/./python',

# C++ function `homePythonPath`
'C:/Users/User/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python',
 
# C++ function `homePluginsPath`
# same as `qgis.utils.HOME_PLUGIN_PATH`
'C:/Users/User/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python/plugins',
 
# C++ function `pluginsPath`
# same as `qgis.utils.sys_plugin_path`
'C:/OSGeo4W/apps/qgis-dev/./python/plugins',

 # added by `site` package (this is normally executed at python start)
 'C:\\OSGeo4W\\apps\\grass\\grass83\\etc\\python',
 'C:\\OSGeo4W\\apps\\gdal-dev\\lib\\site-packages',
 'C:\\OSGeo4W\\bin',
 'C:\\OSGeo4W\\bin\\python312.zip',
 'C:\\OSGeo4W\\apps\\Python312\\DLLs',
 'C:\\OSGeo4W\\apps\\Python312\\Lib',
 'C:\\OSGeo4W\\apps\\Python312',
 'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages',
 'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\win32',
 'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\win32\\lib',
 'C:\\OSGeo4W\\apps\\Python312\\Lib\\site-packages\\Pythonwin',
 'C:/Users/User/AppData/Roaming/QGIS/QGIS3\\profiles\\default/python' 
 ]

Path where qgis module is located:

>>> qgis.__path__
['C:\\OSGeo4W/apps/qgis-dev/./python\\qgis']

The difference clearly shows that path for qgis is added after PYQGIS_STARTUP script is executed. Here is code:

QGIS/src/python/qgspythonutilsimpl.cpp

Lines 103 to 144 in da1f55b

	runString( QStringLiteral( "is_startup_script_executed = run_startup_script(pyqgstart)" ) );
	#ifdef Q_OS_WIN
	runString( "oldhome=None" );
	runString( "if 'HOME' in os.environ: oldhome=os.environ['HOME']\n" );
	runString( "os.environ['HOME']=os.environ['USERPROFILE']\n" );
	#endif
	// construct a list of plugin paths
	// plugin dirs passed in QGIS_PLUGINPATH env. variable have highest priority (usually empty)
	// locally installed plugins have priority over the system plugins
	// use os.path.expanduser to support usernames with special characters (see #2512)
	QStringList pluginpaths;
	const QStringList extraPaths = extraPluginsPaths();
	for ( const QString &path : extraPaths )
	{
	QString p = path;
	if ( !QDir( p ).exists() )
	{
	QgsMessageOutput *msg = QgsMessageOutput::createMessageOutput();
	msg->setTitle( QObject::tr( "Python error" ) );
	msg->setMessage( QObject::tr( "The extra plugin path '%1' does not exist!" ).arg( p ), QgsMessageOutput::MessageText );
	msg->showMessage();
	}
	#ifdef Q_OS_WIN
	p.replace( '\\', "\\\\" );
	#endif
	// we store here paths in unicode strings
	// the str constant will contain utf8 code (through runString)
	// so we call '...'.decode('utf-8') to make a unicode string
	pluginpaths << '"' + p + '"';
	}
	pluginpaths << homePluginsPath();
	pluginpaths << '"' + pluginsPath() + '"';
	// expect that bindings are installed locally, so add the path to modules
	// also add path to plugins
	QStringList newpaths;
	newpaths << '"' + pythonPath() + '"';
	newpaths << homePythonPath();
	newpaths << pluginpaths;
	runString( "sys.path = [" + newpaths.join( QLatin1Char( ',' ) ) + "] + sys.path" );

---

According to docs, this is intentional, so we cannot change the order of these lines.
I think the best option is to queue these messages, and later after we stop changing path, and we are sure that qgis module is imported, then log them using QgsMessageLog.

---

I am going to prepare a pull request in next days (probably tomorrow, max 3 days)