SQL updates for the `planes` table #1211

chrisrosset · 2023-06-23T01:26:57Z

This updates create.sql with the schema changes that were brought in through clean-duplicate-planes.sql in 36b3ee9.

Additionally, this updates load-data.sql to seed the DB with the data present in data/planes.dat.

Resolves #1210.

This updates `create.sql` with the schema changes that were brought in through `clean-duplicate-planes.sql` in jpatokal@36b3ee9. Additionally, this updates `load-data.sql` to seed the DB with the data present in `data/planes.dat`. Resolves jpatokal#1210.

chrisrosset · 2023-06-23T01:40:26Z

@jpatokal If you ever get a moment, perhaps you could share a schema dump (I'm not a MySQL expert but I think mysqldump --no-data flightdb2 would do the trick) of all the tables in the database?

I remember @reedy updated create.sql for countries, airports and airlines based on the schema you shared in #1137 (comment). I would gladly update all the other ones.

chrisrosset · 2023-06-25T23:51:01Z

I just noticed that #1148 did not include the frequency column from #1137 (comment). @reedy, was this on purpose?

reedy · 2023-06-26T11:34:58Z

I just noticed that #1148 did not include the frequency column from #1137 (comment). @reedy, was this on purpose?

Not particularly. Probably mostly because it wasn't throwing an error...

I'd be curious if it's even used, as it seems to be (maybe) updated uses a script that pulls it in from VRS - https://github.com/jpatokal/openflights/blob/2f86a25d09c0721a2b37aa464bd7c474244fdeab/tools/import-vrs.py

reedy · 2023-06-26T11:39:14Z

Though as per #1148 "create.sql: Integrate *-upgrade.sql", not "update to match live schema"...

https://github.com/jpatokal/openflights/blob/4ff8baf/sql/airlines-upgrade.sql doesn't have a frequency column.

chrisrosset · 2023-06-26T13:08:09Z

Though as per #1148 "create.sql: Integrate *-upgrade.sql", not "update to match live schema"...

Fair enough.

The clean-duplicate-*.sql scripts bring in more columns (iata, icao, and frequency):

https://github.com/jpatokal/openflights/blob/master/sql/clean-duplicate-airlines.sql#L28
https://github.com/jpatokal/openflights/blob/master/sql/clean-duplicate-planes.sql#L33

I'd be curious if it's even used, as it seems to be (maybe) updated uses a script that pulls it in from VRS - https://github.com/jpatokal/openflights/blob/2f86a25d09c0721a2b37aa464bd7c474244fdeab/tools/import-vrs.py

I don't know if they're kept up to date but they are read by the code and produce SQL errors when not present:

openflights/php/import.php

Line 169 in ea99f97

$sql = "select name,alias,alid from airlines where ((name like ? or alias like ?) and (iata != '')) or (name = ?) order by frequency desc;";
openflights/php/autocomplete.php

Line 191 in ea99f97

"(SELECT name,plid FROM planes WHERE " . $query . " AND iata IS NULL ORDER BY frequency DESC LIMIT 6) LIMIT 6";

reedy · 2023-06-26T13:31:06Z

import wasn't broken (at least, not with an SQL error related to this) on my dev wiki, even without (presumably) having the frequency column (I have just checked my live schema, and it's indeed not there, so not apparently added by a random patch I applied).

chrisrosset · 2023-06-26T14:07:01Z

import wasn't broken (at least, not with an SQL error related to this) on my dev wiki, even without (presumably) having the frequency column (I have just checked my live schema, and it's indeed not there, so not apparently added by a random patch I applied).

This code only runs when the import is trying to match names based on strings. I don't know if you saw #1197 (comment) but I'm guessing you were using an import file with ID columns populated by the export/backup.

Here's an example SQL warning:

PHP Warning: PDOStatement::execute(): SQLSTATE[42S22]: Column not found: 1054 Unknown column 'frequency' in 'order clause' in /var/www/openflights/php/import.php on line 172

reedy · 2023-06-26T15:33:16Z

import wasn't broken (at least, not with an SQL error related to this) on my dev wiki, even without (presumably) having the frequency column (I have just checked my live schema, and it's indeed not there, so not apparently added by a random patch I applied).

This code only runs when the import is trying to match names based on strings. I don't know if you saw #1197 (comment) but I'm guessing you were using an import file with ID columns populated by the export/backup.

Anything running the "newer" code would indeed have the column populated.

I don't know from what version of openflights I would've originally exported the backup from.

But TLDR is that I was using my "live" data from openflights.org, into my dev install.

Makes it easy to compare changes...

I probably need to wipe the entire database anyway

jpatokal · 2023-06-28T09:33:25Z

@chrisrosset The frequency column is not used by the live website, IIRC it was there to try to sort out duplicates. Here's what the table looks like on the live system:

mysql> describe planes;
+-----------+-------------+------+-----+---------+----------------+
| Field     | Type        | Null | Key | Default | Extra          |
+-----------+-------------+------+-----+---------+----------------+
| name      | varchar(80) | YES  | UNI | NULL    |                |
| abbr      | text        | YES  |     | NULL    |                |
| speed     | double      | YES  |     | NULL    |                |
| plid      | int(11)     | NO   | PRI | NULL    | auto_increment |
| public    | char(1)     | YES  |     | N       |                |
| iata      | text        | YES  |     | NULL    |                |
| icao      | text        | YES  |     | NULL    |                |
| frequency | int(11)     | YES  |     | 0       |                |
+-----------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

reedy · 2023-06-28T09:51:24Z

sql/create.sql

-  PRIMARY KEY  (`plid`)
+  `iata` text default NULL,
+  `icao` text default NULL,
+  `frequency` int default 0,


looks like this should be int(11) to match the live schema

reedy · 2023-06-28T09:51:54Z

sql/create.sql

@@ -155,12 +157,16 @@ CREATE TABLE `locales` (

 DROP TABLE IF EXISTS `planes`;
 CREATE TABLE `planes` (
-  `name` text,
+  `name` varchar(80),
  `abbr` text,
  `speed` double default NULL,
  `plid` int(11) NOT NULL auto_increment,
  `public` char(1) default NULL,


default 'n'?

chrisrosset · 2023-06-28T12:50:24Z

@jpatokal, many thanks for posting the schema.

@chrisrosset The frequency column is not used by the live website, IIRC it was there to try to sort out duplicates. Here's what the table looks like on the live system:

The columns are used! I've even already posted the links to where they're used above. :) I'll list them in more detail now:

Airline frequency is used during import to sort airlines when matching by name.

$sql = "select name,alias,alid from airlines
where ((name like ? or alias like ?) and (iata != '')) or (name = ?) order by frequency

Plane frequency is used for autocomplete. The code and the comment both reference frequency.

 // Autocompletion for plane types
 // First match against major types with IATA codes, then pad to max 6 by matching against frequency of use
$query = $_POST['plane'];
$name = "%$query%";
$query = "(name LIKE :name OR iata LIKE :name) ";
$sql = "(SELECT name,plid FROM planes WHERE " . $query . " AND iata IS NOT NULL ORDER BY name LIMIT 6) UNION " .
"(SELECT name,plid FROM planes WHERE " . $query . " AND iata IS NULL ORDER BY frequency DESC LIMIT 6) LIMIT 6";

Based on them being used, I think we should include them in create.sql. I'll update this PR to match the live schemas posted here and in the previous PR.

It'd be nice to get a mysqldump --no-data to confirm we haven't missed anything. :)

reedy · 2023-06-28T13:20:16Z

I'd be curious if any indexes are out of line too

Reference jpatokal#1211 (comment).

chrisrosset · 2023-07-08T13:57:32Z

@reedy, anything else here? If we do get a full schema dump, I can do another pass in a separate PR. I'd like to get this merged because the Docker PR (#1206) has this changed merged in already.

reedy

There's a little bit of change in the "functionality" of create.sql now... Not that I'm against it.

But I think

openflights/sql/README

Line 4 in 52e054d

Create empty OpenFlights database schema

and maybe

openflights/widget/index.html

Lines 48 to 50 in 52e054d

    
           <p>Once you have these set up, the <tt>sql/create.sql</tt> script can be used to generate the OpenFlights 
        
           database schema and the <tt>sql/load_data.sql</tt> script can be used to load in the standard 
        
           airport, airline and route databases:</p>

should reference that it is destructive (I know it already was, as it would drop each table)

sql/create.sql

chrisrosset · 2023-07-08T16:57:02Z

and maybe

openflights/widget/index.html

Lines 48 to 50 in 52e054d

<p>Once you have these set up, the <tt>sql/create.sql</tt> script can be used to generate the OpenFlights

database schema and the <tt>sql/load_data.sql</tt> script can be used to load in the standard

airport, airline and route databases:</p>

should reference that it is destructive (I know it already was, as it would drop each table)

I think the widget stuff is all dead code. I opened #1224 to ask @jpatokal about removing it.

chrisrosset mentioned this pull request Jun 23, 2023

Added distinct plane type count #992

Open

reedy reviewed Jun 28, 2023

View reviewed changes

Updates to match live schema

33a81b8

Reference jpatokal#1211 (comment).

reedy requested changes Jul 8, 2023

View reviewed changes

sql/create.sql Outdated Show resolved Hide resolved

sql/create.sql Outdated Show resolved Hide resolved

Review updates

760d339

chrisrosset requested a review from reedy July 8, 2023 16:55

reedy merged commit 2c389c1 into jpatokal:master Jul 9, 2023

chrisrosset deleted the planes-sql-changes branch July 9, 2023 17:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL updates for the `planes` table #1211

SQL updates for the `planes` table #1211

chrisrosset commented Jun 23, 2023

chrisrosset commented Jun 23, 2023

chrisrosset commented Jun 25, 2023

reedy commented Jun 26, 2023 •

edited

Loading

reedy commented Jun 26, 2023

chrisrosset commented Jun 26, 2023

reedy commented Jun 26, 2023 •

edited

Loading

chrisrosset commented Jun 26, 2023 •

edited

Loading

reedy commented Jun 26, 2023

jpatokal commented Jun 28, 2023

reedy Jun 28, 2023

reedy Jun 28, 2023

chrisrosset commented Jun 28, 2023

reedy commented Jun 28, 2023

chrisrosset commented Jul 8, 2023

reedy left a comment

chrisrosset commented Jul 8, 2023

	<p>Once you have these set up, the <tt>sql/create.sql</tt> script can be used to generate the OpenFlights
	database schema and the <tt>sql/load_data.sql</tt> script can be used to load in the standard
	airport, airline and route databases:</p>

SQL updates for the planes table #1211

SQL updates for the planes table #1211

Conversation

chrisrosset commented Jun 23, 2023

chrisrosset commented Jun 23, 2023

chrisrosset commented Jun 25, 2023

reedy commented Jun 26, 2023 • edited Loading

reedy commented Jun 26, 2023

chrisrosset commented Jun 26, 2023

reedy commented Jun 26, 2023 • edited Loading

chrisrosset commented Jun 26, 2023 • edited Loading

reedy commented Jun 26, 2023

jpatokal commented Jun 28, 2023

reedy Jun 28, 2023

Choose a reason for hiding this comment

reedy Jun 28, 2023

Choose a reason for hiding this comment

chrisrosset commented Jun 28, 2023

reedy commented Jun 28, 2023

chrisrosset commented Jul 8, 2023

reedy left a comment

Choose a reason for hiding this comment

chrisrosset commented Jul 8, 2023

SQL updates for the `planes` table #1211

SQL updates for the `planes` table #1211

reedy commented Jun 26, 2023 •

edited

Loading

reedy commented Jun 26, 2023 •

edited

Loading

chrisrosset commented Jun 26, 2023 •

edited

Loading