Add support for windows build #122

Shelnutt2 · 2020-12-07T14:27:53Z

Add support for windows build, requires MariaDB 10.5

ihnorton · 2021-09-10T20:47:18Z

Sync'ing up some discussion about this w/ @bhadhy:

So, I reran the CI for this PR: #122
And I get a similar error about curl, except it fails earlier in the cmake detection step. One possibility is that the base directory path is too long. Another is that using TileDB itself as an externalproject does not quite work to build the sub-dependencies such as curl — I’ve never tried a build like that.

Could you try commenting/removing this line: https://github.com/TileDB-Inc/TileDB-MariaDB/blob/master/CMakeLists.txt#L67
And then do a clean cmake setup and build.

^ that will disable S3, which is the project that (immediately) needs curl. There might be another dependency in Azure, which you could also remove — let’s try to get a successful build going with minimal dependencies, and then we can figure out the problem with the deps.

bhadhy · 2021-09-13T02:04:48Z

Using the 0.10.0 release tag of TileDB-MariaDB, and turing off S3 and Azure build completes successfully.
The steps are:

mkdir build_deps
cd build_deps
powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip"
powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip"

unzip bison-2.4.1-dep.zip
unzip bison-2.4.1-bin.zip

git clone https://github.com/MariaDB/server.git -b 10.5 10.5
cd 10.5

git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile

cd storage\mytile

git checkout tags/0.10.0 -b 0.10.0-branch

cd ....

git submodule update --init --recursive

Modify storage/mytile/CMakeLists.txt, -DTILEDB_S3=OFF, -DTILEDB_AZURE=OFF

mkdir builddir
cd builddir

cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -SWITH_DEBUG=1 ..

cmake --build . 2>&1 | tee cmake.build.log

If I turn on S3, the build complains about long path names and fails.
If I turn on Azure, there are additional issues:

C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(212,45): error C3861: 'runtime_error': identifier not found
It can be solved by adding #include stdexcept to base64.cpp
However link fails looking for unresolved external symbol __imp___CrtDbgReportw

ihnorton · 2021-09-13T02:14:14Z

However link fails looking for unresolved external symbol __imp___CrtDbgReportw

This means there is a mismatch between the build types of some projects (eg some projects in debug mode, others release). I believe what's happening is the mytile/mariadb project is configured with -DCMAKE_BUILD_TYPE=Debug, but that setting is not passed in to the libtiledb externalproject. So libtiledb is building in Release. You could do either of these options:

use -DCMAKE_BUILD_TYPE=Release for the mytile project
add -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} on one of these lines of the mytile externalproject config for libtiledb. (for example how this should look, see this line in the main tiledb repo)

ihnorton · 2021-09-13T02:15:21Z

(the 2nd option is better for long-term maintainability, so would suggest making a PR with that change at some point - if this analysis is correct)

bhadhy · 2021-09-14T01:33:32Z

Regarding inclusion of stdexcept to base64.cpp. The https://github.com/Azure/azure-storage-cpplite/blob/master/src/base64.cpp already has the fix. However we are pulling 0.3.0 tag. And there is no newer tag. What do we do?

bhadhy · 2021-09-14T01:46:40Z

Regarding release vs debug build:
I made the change you suggested to CMakeLists.txt by adding
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}

The release build from top level built successfully. The steps were:
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release ..

cmake --build . --config Release 2>&1 | tee cmake.build.log

However doing the similar thing for debug ends up in
cl /c /IC:\work\mariadb\10.5\builddir\include /IC:\work\mariadb\10.5\include /IC:\work\mariadb\10.5\sql /I"C:\work\mariadb\10.5\builddir\extra\pcre2\src\pcre2-build" /IC:\work\mariadb\10.5\builddir\extra\pcre2\src\pcre2\src /IC:\work\mariadb\10.5\builddir\extra\wolfssl /IC:\work\mariadb\10.5\extra\wolfssl\wolfssl /IC:\work\mariadb\10.5\extra\wolfssl\wolfssl\wolfssl /IC:\work\mariadb\10.5\zlib /IC:\work\mariadb\10.5\builddir\zlib /IC:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include /IC:\work\mariadb\10.5\storage\mytile\mytile /Zi /W3 /WX- /diagnostics:caret /MP /Od /Ob0 /D WIN32 /D _WINDOWS /D ENABLED_DEBUG_SYNC /D MYSQL_DYNAMIC_PLUGIN /D WIN /D _CRT_SECURE_NO_DEPRECATE /D _WIN32_WINNT=0x0A00 /D NOMINMAX /D WIN32_LEAN_AND_MEAN /D _CRT_NONSTDC_NO_WARNINGS /D DBUG_TRACE /D HAVE_CONFIG_H /D "CMAKE_INTDIR="Debug"" /D mytile_EXPORTS /D _WINDLL /D _MBCS /Gm- /EHsc /RTC1 /MT /GS /fp:precise /permissive- /Zc:wchar_t /Zc:forScope /Zc:inline /GR /std:c++14 /Fo"mytile.dir\Debug\" /Fd"mytile.dir\Debug\vc142.pdb" /Gd /TP /errorReport:queue /we4099 /we4700 /we4311 /we4477 /we4302 /we4090 /d2OptimizeHugeFunctions -Wno-error=non-virtual-dtor C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-discovery.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-metadata.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-range.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-sysvars.cc" C:\work\mariadb\10.5\storage\mytile\mytile\mytile.cc C:\work\mariadb\10.5\storage\mytile\mytile\utils.cc
cl : command line error D8021: invalid numeric argument '/Wno-error=non-virtual-dtor' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

bhadhy · 2021-09-14T02:19:35Z

CMakeLists.txt has following line.
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")

If I comment that out, I get following error
C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: throw std::runtime_error("Attempting to set type with incorrect size."); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: ^ (compiling source file C:\work\mariadb\10.5\storage\mytile\mytile\utils.cc) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

So I added #include stdexcept to type.h

Next I get following error
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: '#': invalid character: possibly the result of a macro expansion [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: unexpected #else [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: #else [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

So I changed it to:
ulong tile::mytile::index_flags(uint idx, uint part, bool all_parts) const {
DBUG_ENTER("tile::mytile::index_flags");
#if MYSQL_VERSION_ID < 100500
DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
HA_DO_INDEX_COND_PUSHDOWN);
#else
DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
HA_DO_INDEX_COND_PUSHDOWN | HA_CLUSTERED_INDEX);
#endif
}

Now I am getting linker error
LINK : fatal error LNK1104: cannot open file 'externals\install\lib\tiledb.dll.2.4' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

ihnorton · 2021-09-14T02:30:59Z

CMakeLists.txt has following line.
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")

This option is gcc-only, so it needs to be guarded like this:

if(MSVC)
  set(CMAKE_CXX_FLAGS ...)
endif()

Also, just to double-check: when you run the cmake build with CMAKE_BUILD_TYPE=Debug, is it in a clean directory? If not, please try to start in a clean build directory (MSVC tries to separate debug vs release but it often doesn't really work with inter-dependent projects)

bhadhy · 2021-09-16T16:38:45Z

I changed storage/mytile/CMakeLists.txt to have MSVC guard as you suggested:

    if(NOT MSVC)
        SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")    
    endif()

Then I also changed the TILEDB_LIBRARIES for MSVC so that it points to .lib instead of .dll.2.4

    if(MSVC)
        SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}")
    else()
        SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}")
    endif()        
    message(STATUS "TILEDB_LIBRARIES: ${TILEDB_LIBRARIES}")

Now the error I am seeing is following:

mytile-metadata.obj : error LNK2001: unresolved external symbol "class Time_zone * my_tz_OFFSET0" (?my_tz_OFFSET0@@3PEAVTime_zone@@EA) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol "class Time_zone * my_tz_OFFSET0" (?my_tz_OFFSET0@@3PEAVTime_zone@@EA) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_ucs2_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_utf16_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_utf32_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\builddir\storage\mytile\Debug\ha_mytile.dll : fatal error LNK1120: 4 unresolved externals [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

Google search makes me think that these symbols are defined in MYSQL. How do we add mysql library to the list of libraries to be linked?

bhadhy · 2021-09-16T16:45:55Z

Do you know what is "-SWITH_DEBUG=1"?
It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml

bhadhy · 2021-09-16T17:16:44Z

Once again in storage/mytile/CMakeLists.txt, I dropped the MODULE_ONLY

if(WITH_EMBEDDED_SERVER)
    MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
# Else build as a dynamic library
else()
#    MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT MODULE_ONLY RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
    MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
endif()

Now the build goes past the link.
It ends in following error while creating initial database:

Hardlink created for mysqld.exe <<===>> mariadbd.exe
  Generating initdb.dep
  Executing C:/work/mariadb/10.5/builddir/sql/Debug/mariadbd.exe --no-defaults --console --bootstrap --datadir=.
  input file bootstrap.sql, current directory C:/work/mariadb/10.5/builddir/sql/data
  CMake Error at C:/work/mariadb/10.5/cmake/create_initial_db.cmake:68 (MESSAGE):
    Could not create initial database
  
      
     
  
  
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(238,5): error MSB8066: Custom build for 'C:\work\mariadb\10.5\builddir\CMakeFiles\bcccb9ef86ddfbdefa040957b508ecb6\initdb.dep.rule;C:\work\mariadb\10.5\builddir\CMakeFiles\af6190842cf923122927623d5a3c966f\initial_database.rule' exited with code 1. [C:\work\mariadb\10.5\builddir\sql\initial_database.vcxproj]

ihnorton · 2021-09-16T17:18:31Z

Do you know what is "-SWITH_DEBUG=1"?
It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml

That sets whether to build in debug mode I guess, but it looks like it only applies to gcc: https://github.com/MariaDB/server/blob/64f7dffcc7e0e69c31d9a36c2090a26300e57c4c/storage/mroonga/CMakeLists.txt#L322-L330

ihnorton · 2021-09-16T17:19:18Z

Now the build goes past the link.

Good news, hopefully!

It ends in following error while creating initial database:

Is this part of the test suite?

bhadhy · 2021-09-17T01:57:50Z

Do you know what is "-SWITH_DEBUG=1"?
It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml

That sets whether to build in debug mode I guess, but it looks like it only applies to gcc: https://github.com/MariaDB/server/blob/64f7dffcc7e0e69c31d9a36c2090a26300e57c4c/storage/mroonga/CMakeLists.txt#L322-L330

So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?

ihnorton · 2021-09-17T02:28:22Z

So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?

Ah - it should be -DWITH_DEBUG if you want to set it. I don't think -SWITH_DEBUG would have any effect (it might even break something b/c -S has a different meaning for cmake)

Shelnutt2 · 2021-09-17T02:39:21Z

So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?

Ah - it should be -DWITH_DEBUG if you want to set it. I don't think -SWITH_DEBUG would have any effect (it might even break something b/c -S has a different meaning for cmake)

-SWITH_DEBUG is listed in the docs: https://mariadb.com/docs/reference/mdb/system-variables/debug/

ihnorton · 2021-09-17T02:49:03Z

-SWITH_DEBUG is listed in the docs: https://mariadb.com/docs/reference/mdb/system-variables/debug/

Far be it for me to argue with their docs, but I don't see how that would work. AFACT it's a cmake option here, and cmake -S sets the source path:

Usage

  cmake [options] <path-to-source>
  cmake [options] <path-to-existing-build>
  cmake [options] -S <path-to-source> -B <path-to-build>

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Options
  -S <path-to-source>          = Explicitly specify a source directory.
  -B <path-to-build>           = Explicitly specify a build directory.
...

elsewhere it's documented as -DWITH_DEBUG: https://mariadb.com/kb/en/server-system-variables/#debugdebug_dbug

(that said, doesn't seem like the main problem here)

bhadhy · 2021-09-21T14:00:11Z

I am sorry. Looks like the same problems exists in both Release or Debug builds.
I started from a completely clean folder to do a release build and encountered the same set of issues as I did for the debug build.
Here are the steps I followed.

Set environment

@echo off
set PATH=C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;%PATH%
set PATH=C:\work\utils\strawberry-perl-5.32.1.1-64bit\perl\bin;%PATH%
set PATH=C:\work\utils\diffutils-2.8.7-1-bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;%PATH%

Add build dependencies

mkdir build_deps
cd build_deps
powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip"
powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip"
unzip bison-2.4.1-dep.zip
unzip bison-2.4.1-bin.zip

set PATH=%PATH%;C:\work\mariadb\build_deps\bin

checkout mariadb and tiledb-mariadb

git clone https://github.com/MariaDB/server.git -b 10.5 10.5
cd 10.5
git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile
cd storage\mytile
git checkout tags/0.10.0 -b 0.10.0-branch
cd ..\..
git submodule update --init --recursive

3.5 Make following changes to storage\mytile\CMakeLists.txt

(i) Do not use with MSVC

if(NOT MSVC)
    SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
endif()

(ii) Propogate build type (do not mix debug and release)
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
(iii) Build without AWS
-DTILEDB_S3=OFF
(iv) With MSVC specify static libraries

if(MSVC)
        SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}")
else()
        SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}")
endif()

(v) Remove MODULE_ONLY from MYSQL_ADD_PLUGIN
MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
Or we could probably say -DWITH_EMBEDDED_SERVER

Debug build

mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -DWITH_DEBUG=1 ..
cmake --build . --config Debug 2>&1 | tee cmake.build.log

Release build

mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release 2>&1 | tee cmake.build.log

Bugs:
(i) base64.cpp

C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(108,28): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk-build\azure-storage-lite.vcxproj] [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\ep_azuresdk.vcxproj]
Why it is not being patched?
Need to add 
#include <stdexcept>

(ii) type.h

C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

Need to add
#include <stdexcept>

(iii) ha_mytile.cc

C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059:               HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

Fix it

#if MYSQL_VERSION_ID < 100500
  DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
              HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
              HA_DO_INDEX_COND_PUSHDOWN);
#else
  DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
              HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
              HA_DO_INDEX_COND_PUSHDOWN | HA_CLUSTERED_INDEX);
#endif

(iv) Could not create initial database
Executing C:/work/mariadb/10.5/builddir/sql/Release/mariadbd.exe --no-defaults --console --bootstrap --datadir=.
input file bootstrap.sql, current directory C:/work/mariadb/10.5/builddir/sql/data
CMake Error at C:/work/mariadb/10.5/cmake/create_initial_db.cmake:68 (MESSAGE):
Could not create initial database

bhadhy · 2021-09-23T15:26:07Z

Here are the steps I followed.

Set environment

@echo off
set PATH=C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;%PATH%
set PATH=C:\work\utils\strawberry-perl-5.32.1.1-64bit\perl\bin;%PATH%
set PATH=C:\work\utils\diffutils-2.8.7-1-bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;%PATH%


2. Add build dependencies

mkdir build_deps
cd build_deps
powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip"
powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip"
unzip bison-2.4.1-dep.zip
unzip bison-2.4.1-bin.zip

set PATH=%PATH%;C:\work\mariadb\build_deps\bin

3. checkout mariadb and tiledb-mariadb
git clone https://github.com/MariaDB/server.git -b 10.5 10.5
cd 10.5
git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile
cd storage\mytile
git checkout tags/0.10.0 -b 0.10.0-branch
cd ..\..
git submodule update --init --recursive

3.5 Make following changes to storage\mytile\CMakeLists.txt

(i) Do not use with MSVC
if(NOT MSVC)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
endif()

(ii) Propogate build type (do not mix debug and release)
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
(iii) Build without AWS
-DTILEDB_S3=OFF
(iv) With MSVC specify static libraries
if(MSVC)
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}")
else()
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}")
endif()

(v) Remove MODULE_ONLY from MYSQL_ADD_PLUGIN for both "WITH_EMBEDDED_SERVER" and "not WITH_EMBEDDED_SERVER"
MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
Building by specifying -DWITH_EMBEDDED_SERVER at top level had a lot of issues.

(vi) Copy tiledb.dll to buildir/sql/Release. It is needed by builddir/sql/Release/mariadbd.exe to create initial database. So that we do not have to have the dll in path.
add_custom_command(TARGET tiledb POST_BUILD COMMAND ${CMAKE_COMMAND} -E copy ${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX} ${CMAKE_BINARY_DIR}/sql/Release )

Debug build

mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -DWITH_DEBUG=1 ..
cmake --build . --config Debug 2>&1 | tee cmake.build.log

5. Release build

mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release 2>&1 | tee cmake.build.log

Bugs:

(i) base64.cpp
C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(108,28): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk-build\azure-storage-lite.vcxproj] [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\ep_azuresdk.vcxproj]

Why it is not being patched?
Need to add

#include

(ii) type.h

C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

Need to add

#include

(iii) ha_mytile.cc

C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059:               HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]

Fix it

Create a package

cmake --build . --config Release --target package 2>&1 | tee cmake.package.log
Copy package mariadb-10.5.13-winx64.zip from C:\work\mariadb\10.5\builddir to C:\work\mariadb
Extract the contents to C:\work\mariadb\mariadb-10.5.13-winx64
Manually copy C:\work\mariadb\10.5\builddir\sql\Release\tiledb.dll to C:\work\mariadb\mariadb-10.5.13-winx64\bin

7. Open a new command prompt. Put the installation in path.

set PATH=C:\work\mariadb\mariadb-10.5.13-winx64\bin;%PATH%

Create a default database

$ mysql_install_db.exe --datadir=c:\work\mariadb\db --default-user --password=secret
Running bootstrap
2021-09-23  9:35:59 0 [Note] C:\work\mariadb\mariadb-10.5.13-winx64\bin\mysqld.exe (mysqld 10.5.13-MariaDB) starting as process 31640 ...

Setting root password
Creating my.ini file
Creation of the database was successful

9. Modify the c:\work\mariadb\db\my.ini to following

[mysqld]
port=3306
plugin-maturity=experimental
#plugin-load-add=ha_mytile
#Enable embedded mytile
plugin-mytile=ON
#default-storage-engine=MyTile
mytile_delete_arrays=ON
#default-time-zone=UTC

sql-mode=NO_ENGINE_SUBSTITUTION
explicit-defaults-for-timestamp=1

#The following is to get rid of the harmless
#"Deadlock found when trying to get lock" errors, see MDEV-12285.
log-warnings=1

datadir=c:/work/mariadb/db
[client]
plugin-dir=C:/work/mariadb/mariadb-10.5.13-winx64/lib/plugin

Start the mariadbd server

$ mariadbd.exe --defaults-file=c:\work\mariadb\db\my.ini --console
2021-09-23  9:47:15 0 [Note] mariadbd.exe (mysqld 10.5.13-MariaDB) starting as process 34736 ...
2021-09-23  9:47:15 0 [Note] InnoDB: Uses event mutexes
2021-09-23  9:47:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-09-23  9:47:15 0 [Note] InnoDB: Number of pools: 1
2021-09-23  9:47:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-09-23  9:47:15 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-09-23  9:47:15 0 [Note] InnoDB: Completed initialization of buffer pool
2021-09-23  9:47:15 0 [Note] InnoDB: 128 rollback segments are active.
2021-09-23  9:47:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-09-23  9:47:15 0 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-09-23  9:47:15 0 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2021-09-23  9:47:15 0 [Note] InnoDB: 10.5.13 started; log sequence number 45106; transaction id 20
2021-09-23  9:47:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-09-23  9:47:15 0 [Note] InnoDB: Loading buffer pool(s) from c:\work\mariadb\db\ib_buffer_pool
2021-09-23  9:47:15 0 [Note] Server socket created on IP: '::'.
2021-09-23  9:47:15 0 [Note] InnoDB: Buffer pool(s) load completed at 210923  9:47:15
2021-09-23  9:47:15 0 [Note] mariadbd.exe: ready for connections.
Version: '10.5.13-MariaDB'  socket: ''  port: 3306  Source distribution
2021-09-23  9:55:38 4 [Note] TILEDB_POSITIVE_DELTA_MAX_WINDOW=128

11. Open another command prompt.  Put the installation in path.

set PATH=C:\work\mariadb\mariadb-10.5.13-winx64\bin;%PATH%

Use mariadb to connect to server and test

$ mariadb --user=root --password=secret                                                                             
Welcome to the MariaDB monitor.  Commands end with ; or \g.                                                         
Your MariaDB connection id is 4                                                                                     
Server version: 10.5.13-MariaDB Source distribution                                                                 
                                                                                                                    
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.                                                
                                                                                                                    
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.                                      
                                                                                                                    
MariaDB [(none)]> CREATE DATABASE bookstore;
Query OK, 1 row affected (0.003 sec)                                                                                
                                                                                                                    
MariaDB [(none)]> USE bookstore;
Database changed                                                                                                    
MariaDB [bookstore]> create table test_array (   dim0 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10",   dim1 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10",   attr1 varchar(255) filters="GZIP=-1" ) engine=MyTile array_type='SPARSE' coordinate_filters="NONE" offset_filters="POSITIVE_DELTA=128";      
Query OK, 0 rows affected (0.021 sec)                                                                               
                                                                                                                    
MariaDB [bookstore]> insert into  test_array (dim0, dim1, attr1) values (1, 1, "cell 1"), (1, 2, "cell 2"), (2, 1, "cell 3");
Query OK, 3 rows affected (0.036 sec)                                                                               
Records: 3  Duplicates: 0  Warnings: 0                                                                              
                                                                                                                    
MariaDB [bookstore]> select * from test_array;
+------+------+--------+                                                                                            
| dim0 | dim1 | attr1  |                                                                                            
+------+------+--------+                                                                                            
|    1 |    1 | cell 1 |                                                                                            
|    1 |    2 | cell 2 |                                                                                            
|    2 |    1 | cell 3 |                                                                                            
+------+------+--------+                                                                                            
3 rows in set (0.012 sec)                                                                                           
                                                                                                                    
MariaDB [bookstore]>

Shelnutt2 · 2021-09-23T16:40:09Z

@bhadhy great job!! Awesome to see it working on windows!

niclar · 2024-09-19T15:52:28Z

@DimitrisStaratzis any updates on the windows build ?

Shelnutt2 force-pushed the erica/ch922/windows-build branch 6 times, most recently from 4853181 to 5412324 Compare August 19, 2021 14:23

Shelnutt2 force-pushed the erica/ch922/windows-build branch from b596b93 to be8d06d Compare October 2, 2021 14:30

Shelnutt2 and others added 3 commits November 9, 2023 13:23

Set MariaDB version to 10.5 for CI/Docker

09501e2

windows build stuff - current issues linking

16d15b0

fix directory structure

d961c43

Shelnutt2 and others added 5 commits November 9, 2023 13:24

Set windows CI timeout to 90 minutes

d19f4bf

Collect failed build dir from windows CI job

4b54278

WIP try switching build type to release

2bfa3b4

Switch CMakeLists.txt to handle static tiledb generically

00812a2

Remove debug cmake setting for windows, its unused

42858eb

DimitrisStaratzis force-pushed the erica/ch922/windows-build branch from 3299b81 to 42858eb Compare November 9, 2023 11:28

DimitrisStaratzis added 2 commits November 9, 2023 13:38

update MariaDB version to 10.6.10

f392993

install zlib

1fffa83

DimitrisStaratzis force-pushed the erica/ch922/windows-build branch from ded6775 to 525e04f Compare November 9, 2023 17:03

test pure mariadb build

593112a

DimitrisStaratzis force-pushed the erica/ch922/windows-build branch from 525e04f to 593112a Compare November 9, 2023 17:27

DimitrisStaratzis self-assigned this Nov 9, 2023

DimitrisStaratzis force-pushed the erica/ch922/windows-build branch from 9df3787 to 650b072 Compare November 10, 2023 16:00

fix dependency installation

6286aad

DimitrisStaratzis force-pushed the erica/ch922/windows-build branch from 650b072 to 6286aad Compare November 10, 2023 16:03

DimitrisStaratzis and others added 13 commits November 10, 2023 18:28

re-enable TileDB

01a7302

build MariaDB in debug mode

d81696b

turn s3 off for TileDB

07fb425

disable TileDB superbuild.

ca95070

add TileDB to PATH

b7f0933

remove debug mode from TileDB build

ded5a66

add TILEDB_INCLUDE_DIRECTORIES property when using static TileDB

851a7f9

update to TileDB-2.17.4 in azure_pipelines.yml

18b95df

remove preprocessor conditions for mysql < 10.5

c008ae4

re enable preprocessor conditions but now corrected

c1e28d5

try building mariadb in release mode

5d2195b

remove tiledb_install/bin from PATH

f77a223

re-add MODULE_ONLY in CMakeLists.txt for windows build

5e5b07b

jdblischak mentioned this pull request Sep 19, 2024

mytile 0.33.0 windows build error #380

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for windows build #122

Add support for windows build #122

Shelnutt2 commented Dec 7, 2020 •

edited

Loading

ihnorton commented Sep 10, 2021

bhadhy commented Sep 13, 2021 •

edited

Loading

ihnorton commented Sep 13, 2021 •

edited

Loading

ihnorton commented Sep 13, 2021

bhadhy commented Sep 14, 2021

bhadhy commented Sep 14, 2021

bhadhy commented Sep 14, 2021

ihnorton commented Sep 14, 2021

bhadhy commented Sep 16, 2021 •

edited

Loading

bhadhy commented Sep 16, 2021 •

edited

Loading

bhadhy commented Sep 16, 2021

ihnorton commented Sep 16, 2021

ihnorton commented Sep 16, 2021

bhadhy commented Sep 17, 2021

ihnorton commented Sep 17, 2021

Shelnutt2 commented Sep 17, 2021

ihnorton commented Sep 17, 2021

bhadhy commented Sep 21, 2021 •

edited by ihnorton

Loading

bhadhy commented Sep 23, 2021 •

edited

Loading

Shelnutt2 commented Sep 23, 2021

niclar commented Sep 19, 2024

Add support for windows build #122

Are you sure you want to change the base?

Add support for windows build #122

Conversation

Shelnutt2 commented Dec 7, 2020 • edited Loading

ihnorton commented Sep 10, 2021

bhadhy commented Sep 13, 2021 • edited Loading

ihnorton commented Sep 13, 2021 • edited Loading

ihnorton commented Sep 13, 2021

bhadhy commented Sep 14, 2021

bhadhy commented Sep 14, 2021

bhadhy commented Sep 14, 2021

ihnorton commented Sep 14, 2021

bhadhy commented Sep 16, 2021 • edited Loading

bhadhy commented Sep 16, 2021 • edited Loading

bhadhy commented Sep 16, 2021

ihnorton commented Sep 16, 2021

ihnorton commented Sep 16, 2021

bhadhy commented Sep 17, 2021

ihnorton commented Sep 17, 2021

Shelnutt2 commented Sep 17, 2021

ihnorton commented Sep 17, 2021

bhadhy commented Sep 21, 2021 • edited by ihnorton Loading

bhadhy commented Sep 23, 2021 • edited Loading

Shelnutt2 commented Sep 23, 2021

niclar commented Sep 19, 2024

Shelnutt2 commented Dec 7, 2020 •

edited

Loading

bhadhy commented Sep 13, 2021 •

edited

Loading

ihnorton commented Sep 13, 2021 •

edited

Loading

bhadhy commented Sep 16, 2021 •

edited

Loading

bhadhy commented Sep 16, 2021 •

edited

Loading

bhadhy commented Sep 21, 2021 •

edited by ihnorton

Loading

bhadhy commented Sep 23, 2021 •

edited

Loading