Bug Report: vtgates crash on start due to column_format compressed

[rohitnandwani]

Overview of the Issue

Creation of compressed columns (https://docs.percona.com/percona-server/5.7/flexibility/compressed_columns.html) results in a crash (panic) of all vtgates. The vtgates keep crashing on being restarted.

Reproduction Steps

1. Deploy the following schema:

CREATE TABLE `compress_test` (
  `col2` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
  `col3` bigint DEFAULT NULL,
  `col4` longblob /*!50633 COLUMN_FORMAT COMPRESSED */,
  `col5` bigint DEFAULT NULL,
  `col6` tinyint(1) DEFAULT NULL,
  `col7` varbinary(520) DEFAULT NULL,
  `col8` bigint DEFAULT NULL,
  `col1` varbinary(750) DEFAULT NULL,
  `col9` varbinary(20) DEFAULT NULL,
  `col10` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
  `id` bigint unsigned NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `col1` (`col1`),
  UNIQUE KEY `col10` (`col10`),
  KEY `col8` (`col8`),
  KEY `DSKEY_FULL` (`col7`,`col8`,`col5`),
  KEY `PREFIX_KEY` (`col2`,`col3`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED;

2. View error. Restarting vtgates results in the same error again.

3. Cleanup the erroneous table:

vtctldclient ApplySchema --sql "drop table compress_test;" _keyspace_

Binary Version

`vtgate --version`

vtgate version Version: 19.0.8 (Git revision 2665aa2a74fdd26c4606598ba68746be30bd9148 branch 'HEAD') built on Tue Dec  3 05:09:55 UTC 2024 by runner@fv-az1670-25 using go1.22.9 linux/amd64

Operating System and Environment details

`cat /etc/os-release`

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"


`uname -sr`

Linux 3.10.0-1160.80.1.el7.x86_64


`uname -m`

x86_64

Log Fragments

parser.go:85] ignoring error parsing DDL 'CREATE TABLE `compress_test` (
  `col2` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
  `col3` bigint DEFAULT NULL,
  `col4` longblob /*!50633 COLUMN_FORMAT COMPRESSED */,
  `col5` bigint DEFAULT NULL,
  `col6` tinyint(1) DEFAULT NULL,
  `col7` varbinary(520) DEFAULT NULL,
  `col8` bigint DEFAULT NULL,
  `col1` varbinary(750) DEFAULT NULL,
  `col9` varbinary(20) DEFAULT NULL,
  `col10` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
  `id` bigint unsigned NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `col1` (`col1`),
  UNIQUE KEY `col10` (`col10`),
  KEY `col8` (`col8`),
  KEY `DSKEY_FULL` (`col7`,`col8`,`col5`),
  KEY `PREFIX_KEY` (`col2`,`col3`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED': syntax error at position 194 near 'COMPRESSED'
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x160bfb2]

goroutine 1 gp=0xc0000061c0 m=11 mp=0xc001580008 [running]:
panic({0x1a5da00?, 0x3564c70?})
	runtime/panic.go:779 +0x158 fp=0xc00c6cee50 sp=0xc00c6ceda0 pc=0x43dd38
vitess.io/vitess/go/exit.doRecover({0x1a5da00?, 0x3564c70?}, 0x80?)
	vitess.io/vitess/go/exit/exit.go:92 +0x12a fp=0xc00c6ceea8 sp=0xc00c6cee50 pc=0xc0370a
vitess.io/vitess/go/exit.Recover()
	vitess.io/vitess/go/exit/exit.go:69 +0x1f fp=0xc00c6ceed0 sp=0xc00c6ceea8 pc=0xc035bf
panic({0x1a5da00?, 0x3564c70?})
	runtime/panic.go:770 +0x132 fp=0xc00c6cef80 sp=0xc00c6ceed0 pc=0x43dd12
runtime.panicmem(...)
	runtime/panic.go:261
runtime.sigpanic()
	runtime/signal_unix.go:881 +0x378 fp=0xc00c6cefe0 sp=0xc00c6cef80 pc=0x4574f8
vitess.io/vitess/go/vt/vtgate/schema.getTableCollation(0x18be750?)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:353 +0x12 fp=0xc00c6cf040 sp=0xc00c6cefe0 pc=0x160bfb2
vitess.io/vitess/go/vt/vtgate/schema.getColumns(0x0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:314 +0x2a fp=0xc00c6cf190 sp=0xc00c6cf040 pc=0x160bc6a
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).updateTables(0xc00c543200, {0xc00c53cde0, 0x9}, 0xc00c937200?)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:307 +0xf3 fp=0xc00c6cf2f8 sp=0xc00c6cf190 pc=0x160b773
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).loadTables.func1(0xc00c937200)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:111 +0x38 fp=0xc00c6cf330 sp=0xc00c6cf2f8 pc=0x1609b58
vitess.io/vitess/go/vt/vttablet/grpctabletconn.(*gRPCQueryClient).GetSchema(0xc00a69c540, {0x22b6798?, 0x365b120?}, 0xc00c628e40, 0x1, {0x0, 0x0, 0x0}, 0xc00c46d520)

[derekperkins]

I triggered this same error today with a generated point column. Here was the alter schema I ran, which successfully was processed through a vtgate connection.

alter table locations
add lat_long point as (
    point(
            geocode->>'$.geometry.location.lat',
            geocode->>'$.geometry.location.lng'
    )
)
SRID 4326 after geocodes;

Here was the sql that came through in the vtgate logs before the panic

W0131 22:04:23.457072       1 parser.go:85] ignoring error parsing DDL 'CREATE TABLE `locations` (
  `location_id` mediumint unsigned NOT NULL AUTO_INCREMENT,
  `country_code` binary(2) NOT NULL,
  `type` varbinary(22) NOT NULL,
  `location` varbinary(85) NOT NULL,
  `name_no_context` varbinary(85) NOT NULL,
  `country_location_id` mediumint unsigned NOT NULL,
  `geocode` json DEFAULT NULL,
  `geocodes` json DEFAULT NULL,
  `lat_long` point GENERATED ALWAYS AS (point(json_unquote(json_extract(`geocode`,_utf8mb4'$.geometry.location.lat')),json_unquote(json_extract(`geocode`,_utf8mb4'$.geometry.location.lng')))) VIRTUAL /*!80003 SRID 4326 */,
  `time_zone` json DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`location_id`),
  UNIQUE KEY `country_code__type__location__uq` (`country_code`,`type`,`location`),
  UNIQUE KEY `location_id__country_code__uq` (`location_id`,`country_code`),
  KEY `name_no_context__type` (`name_no_context`,`type`),
  KEY `type__name_no_context` (`type`,`name_no_context`),
  KEY `location__type` (`location`,`type`),
  KEY `type__location` (`type`,`location`),
  KEY `locations__country_location_id__location_id__fk` (`country_location_id`),
  CONSTRAINT `fk_Locations_Countries` FOREIGN KEY (`country_code`) REFERENCES `countries` (`country_code`),
  CONSTRAINT `locations__country_location_id__location_id__fk` FOREIGN KEY (`country_location_id`) REFERENCES `locations` (`location_id`)
) ENGINE=InnoDB AUTO_INCREMENT=314105 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci': syntax error at position 576 near 'SRID'

and the vtgate panic itself

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x1724e12]

goroutine 1 [running]:
vitess.io/vitess/go/exit.doRecover({0x1b80f40?, 0x378f650?}, 0x40?)
	vitess.io/vitess/go/exit/exit.go:92 +0x126
vitess.io/vitess/go/exit.Recover()
	vitess.io/vitess/go/exit/exit.go:69 +0x1f
panic({0x1b80f40?, 0x378f650?})
	runtime/panic.go:785 +0x132
vitess.io/vitess/go/vt/vtgate/schema.getTableCollation(0x19d0a2a?)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:413 +0x12
vitess.io/vitess/go/vt/vtgate/schema.getColumns(0x0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:374 +0x2a
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).updateTables(0xc000dddc20, {0xc000bc031c, 0x9}, 0xc00134ccd0?)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:367 +0xf3
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).loadTables.func1(0xc00134ccd0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:120 +0x38
vitess.io/vitess/go/vt/vttablet/grpctabletconn.(*gRPCQueryClient).GetSchema(0xc000921080, {0x2406a68?, 0x3846c20?}, 0xc0012de7e0, 0x1, {0x0, 0x0, 0x0}, 0xc001109640)
	vitess.io/vitess/go/vt/vttablet/grpctabletconn/conn.go:1141 +0x218
vitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).GetSchema.func1({0x2406a68, 0x3846c20}, 0xc001139f38?, {0x242d050?, 0xc000921080?})
	vitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:345 +0x50
vitess.io/vitess/go/vt/vtgate.(*TabletGateway).withRetry(0xc0000f6700, {0x2406a68, 0x3846c20}, 0xc0012de7e0, {0xc000b636a0?, 0x410be5?}, {0x30?, 0x1cad6a0?}, 0x0, 0xc000c38750)
	vitess.io/vitess/go/vt/vtgate/tabletgateway.go:411 +0x872
vitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).GetSchema(0xc000714618, {0x2406a68, 0x3846c20}, 0xc0012de7e0, 0x1, {0x0, 0x0, 0x0}, 0xc001109640)
	vitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:344 +0xf9
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).loadTables(0xc000dddc20, {0x242ce30, 0xc0000f6700}, 0xc0012de7e0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:119 +0x1bb
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).LoadKeyspace(0xc000dddc20, {0x242ce30, 0xc0000f6700}, 0xc0012de7e0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:86 +0x2b
vitess.io/vitess/go/vt/vtgate/schema.(*Tracker).AddNewKeyspace(0xc000dddc20, {0x242ce30, 0xc0000f6700}, 0xc0012de7e0)
	vitess.io/vitess/go/vt/vtgate/schema/tracker.go:482 +0x167
vitess.io/vitess/go/vt/vtgate.resolveAndLoadKeyspace({0x2406b50?, 0xc0002ca370?}, 0xc0002ca370?, 0xc000dddc20, 0xc0000f6700, {0xc000bc031c, 0x9})
	vitess.io/vitess/go/vt/vtgate/vtgate.go:415 +0x29f
vitess.io/vitess/go/vt/vtgate.addKeyspacesToTracker({0x2406b50, 0xc0002ca370}, 0xc000efba40, 0xc000dddc20, 0xc0000f6700)
	vitess.io/vitess/go/vt/vtgate/vtgate.go:396 +0x145
vitess.io/vitess/go/vt/vtgate.Init({0x2406b50, 0xc0002ca370}, 0xc0006e7b90, {0x0?, 0x0?}, {0x240c300, 0xc0004ce160}, {0x7ffc1b62cf10, 0xa}, {0xc000988188, ...}, ...)
	vitess.io/vitess/go/vt/vtgate/vtgate.go:315 +0x9ca

[derekperkins]

Related PRs fixing the panic and adding column compression support:

• Fix panic inside schema tracker #17659
• Handle Percona compressed column extension #17660

I'm not sure what is causing my version of the error, which is a combination of a generated column referencing json and converting it into a geometry point with an SRID, which is surely a pretty small edge case.

[dbussink]

I'm not sure what is causing my version of the error, which is a combination of a generated column referencing json and converting it into a geometry point with an SRID, which is surely a pretty small edge case.

@derekperkins Can you open an issue for this with the schema? It's likely something where we're also missing parser support. But we really want to have complete parsing support, definitely for schema definitions (also for stuff like schemadiff).

[derekperkins]

Opened #17678

[dbussink]

Closing as this has been fixed.