CVE-2015-3152: Update cedar-14 libmysqlclient-dev from 5.5 to 5.7

[edmorley]

The mysql 5.5/5.6 releases of libmysqlclient are still vulnerable to CVE-2015-3152. The distro libmysqlclient-dev Ubuntu 14.04 package is the rather old 5.5.x release, which has absolutely no way to prevent TLS stripping when used by MySQL connectors such as MySQLdb, mysqlclient (see PyMySQL/mysqlclient#98) or even Oracle's own official Connector/Python (when used in non_pure mode; see MySQL bug 82383).

As such (and similar to what is already done for the Postgres client), this switches to using the official MySQL APT source, so we can get the more up to date mysql 5.7 release of libmysqlclient. See:
http://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/#repo-qg-apt-repo-manual-setup

The GPG key was taken from:
http://dev.mysql.com/doc/refman/5.7/en/checking-gpg-signature.html

However to prevent breakage with apps already linked against libmysqlclient.so.18, we still have to install the existing mysql 5.5 based libmysqlclient18 package (which will be sourced from the archive.ubuntu.com repo instead). This is not a problem, since libmysqlclient.so will still end up being symlinked to the newer libmysqlclient.so.20.

Before:

• libmysqlclient-dev 5.5.50-0ubuntu0
• libmysqlclient18 5.5.50-0ubuntu0
• mysql-common 5.5.50-0ubuntu0

After:

• libmysqlclient-dev 5.7.14-1ubuntu14.04
• libmysqlclient18 5.5.50-ubuntu0
• libmysqlclient20 5.7.14-1ubuntu14.04
• mysql-common 5.7.14-1ubuntu14.04

/usr/lib/x86_64-linux-gnu$ ls -l libmysql*
-rw-r--r-- 1 root root 5902058 Jul 12 13:37 libmysqlclient.a
lrwxrwxrwx 1 root root      20 Jul 21 12:58 libmysqlclient_r.so.18 -> libmysqlclient.so.18
lrwxrwxrwx 1 root root      24 Jul 21 12:58 libmysqlclient_r.so.18.0.0 -> libmysqlclient.so.18.0.0
lrwxrwxrwx 1 root root      20 Jul 12 13:37 libmysqlclient.so -> libmysqlclient.so.20
lrwxrwxrwx 1 root root      24 Jul 21 12:58 libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
-rw-r--r-- 1 root root 3355232 Jul 21 12:59 libmysqlclient.so.18.0.0
lrwxrwxrwx 1 root root      24 Jul 12 13:37 libmysqlclient.so.20 -> libmysqlclient.so.20.3.1
-rw-r--r-- 1 root root 4283848 Jul 12 13:37 libmysqlclient.so.20.3.1
-rw-r--r-- 1 root root   18396 Jul 12 13:37 libmysqlservices.a

Fixes:
https://help.heroku.com/tickets/376633

[dzuelke]

This would break all existing apps that use MySQL, as they link against libmysqclient18.

This change should instead install both libmysqlclient18 and libmysqlclient20, and use the newer versions for libmysqlclient-dev and mysql-common.

[edmorley]

I was thinking the caches would just have to be cleared, though:

• this is pretty excessive
• some people might have custom vendored packages (eg in their own S3 buckets) that are already linked against libmysqclient18

That said, installing both libmysqclient18 and libmysqclient20 using apt-get is non-trivial, since libmysqclient20 has a = 5.7.x dependency on mysql-common, whereas libmysqclient18 has = 5.6.x.

As such, one would have to either use the apt-get/dpkg force options (but that leaves apt-get in a broken state for future invocations), or else manually extract libmysqclient18 into the relevant directories (eg with a dpkg -x foo.deb /) and install only libmysqclient20 with apt-get.

Thoughts?

[dzuelke]

I don't think this has anything to do with caches. libmysqlclient.18.so is currently part of the stack image. If you have an application with e.g. the mysql2 Ruby native extension gem installed, then that would simply stop working if that .so is no longer there.

[edmorley]

To be clearer:

1. The stack image contains libmysqlclient, which is currently libmysqlclient.18.so.
2. When a buildpack installs a Python/Ruby/... package that links to libmysqlclient, the result (which directly references libmysqlclient.18.so) is stored in the build cache and only the build cache.
3. If the cache is cleared, the next buildpack build will repeat the compile, and link against whatever is the current libmysqlclient library in the stack image at that point (the packages themselves don't directly reference libmysqlclient.18.so, only libmysqlclient).

Therefore clearing the buildpack cache is sufficient to trigger (3), which will prevent any breakage with packages installed as part of the buildpack build.

However if people use a custom build process (eg bin/post_compile in the Python buildpack) and use that to copy a precompiled binary from their S3 bucket (or wherever) into the app, then that may still break, since no recompilation will have been triggered.

[dzuelke]

That's not how that works though. If the stack image is updated, then it is updated across all runtimes on their next respective instance restart. An application built yesterday, which links against libmysqlclient.so.18 will then be extracted on a runtime that only has libmysqlclient.so.20 when it cycles.

In order for this change to not break all existing apps that use libmysqlclient, each of these apps would need to be re-deployed, and that's not feasible.

[dzuelke]

For the same reason, applications that were deployed to Cedar-10 didn't simply work out of the box on Cedar-14, as they had been compiled against different library versions.

[edmorley]

Ah the slug is decoupled from the stack image, I didn't realise. In which case, yes there is no way to prevent breakage unless libmysqlclient.18.so was still in the image.

I guess that makes sense, otherwise:
(a) you wouldn't be able to roll out eg glibc security updates without getting everyone to perform a new build
(b) the slug archive would be huge

[dzuelke]

Yes exactly!

[dzuelke]

It doesn't appear to be a problem to have both libmysqlclient variants installed:

$ docker run -i -t heroku/cedar:14 /bin/bash

root@cc66c8fd0fda:/# apt-key adv --keyserver pgp.mit.edu --recv-keys 5072E1F5
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.mr1y6AbiX6 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/apt.postgresql.org.gpg --keyserver pgp.mit.edu --recv-keys 5072E1F5
gpg: requesting key 5072E1F5 from hkp server pgp.mit.edu
gpg: key 5072E1F5: public key "MySQL Release Engineering <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1

root@cc66c8fd0fda:/# echo "deb http://repo.mysql.com/apt/ubuntu/ trusty mysql-5.7" > /etc/apt/sources.list.d/mysql.list

root@cc66c8fd0fda:/# apt-get update
Get:1 http://repo.mysql.com trusty InRelease [28.7 kB]
Ign http://archive.ubuntu.com trusty InRelease                                 
Get:2 http://apt.postgresql.org trusty-pgdg InRelease [33.8 kB]                
Get:3 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]            
Get:4 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:5 http://archive.ubuntu.com trusty Release.gpg [933 B]   
Get:6 http://archive.ubuntu.com trusty Release [58.5 kB]
Get:7 http://repo.mysql.com trusty/mysql-5.7 amd64 Packages [2702 B]
Get:8 http://archive.ubuntu.com trusty-security/main amd64 Packages [638 kB]
Get:9 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1100 kB]
Get:10 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:11 http://apt.postgresql.org trusty-pgdg/main amd64 Packages [97.9 kB]
Get:12 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]    
Fetched 11.4 MB in 5s (2219 kB/s)                        
Reading package lists... Done

root@cc66c8fd0fda:/# apt-get install libmysqlclient-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libmysqlclient18
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  libmysqlclient20 mysql-common
The following NEW packages will be installed:
  libmysqlclient20
The following packages will be upgraded:
  libmysqlclient-dev mysql-common
2 upgraded, 1 newly installed, 0 to remove and 429 not upgraded.
Need to get 2078 kB of archives.
After this operation, 6013 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://repo.mysql.com/apt/ubuntu/ trusty/mysql-5.7 mysql-common amd64 5.7.14-1ubuntu14.04 [79.8 kB]
Get:2 http://repo.mysql.com/apt/ubuntu/ trusty/mysql-5.7 libmysqlclient20 amd64 5.7.14-1ubuntu14.04 [827 kB]
Get:3 http://repo.mysql.com/apt/ubuntu/ trusty/mysql-5.7 libmysqlclient-dev amd64 5.7.14-1ubuntu14.04 [1171 kB]
Fetched 2078 kB in 0s (4287 kB/s)        
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48542 files and directories currently installed.)
Preparing to unpack .../mysql-common_5.7.14-1ubuntu14.04_amd64.deb ...
Unpacking mysql-common (5.7.14-1ubuntu14.04) over (5.5.47-0ubuntu0.14.04.1) ...
Selecting previously unselected package libmysqlclient20:amd64.
Preparing to unpack .../libmysqlclient20_5.7.14-1ubuntu14.04_amd64.deb ...
Unpacking libmysqlclient20:amd64 (5.7.14-1ubuntu14.04) ...
Preparing to unpack .../libmysqlclient-dev_5.7.14-1ubuntu14.04_amd64.deb ...
Unpacking libmysqlclient-dev (5.7.14-1ubuntu14.04) over (5.5.47-0ubuntu0.14.04.1) ...
Setting up mysql-common (5.7.14-1ubuntu14.04) ...
update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode
update-alternatives: warning: not replacing /etc/mysql/my.cnf with a link
Setting up libmysqlclient20:amd64 (5.7.14-1ubuntu14.04) ...
Setting up libmysqlclient-dev (5.7.14-1ubuntu14.04) ...
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...

root@cc66c8fd0fda:/# apt list --installed | grep mysql
libmysqlclient-dev/unknown,now 5.7.14-1ubuntu14.04 amd64 [installed]
libmysqlclient18/now 5.5.47-0ubuntu0.14.04.1 amd64 [installed,upgradable to: 5.5.50-0ubuntu0.14.04.1]
libmysqlclient20/unknown,now 5.7.14-1ubuntu14.04 amd64 [installed,automatic]
mysql-common/unknown,now 5.7.14-1ubuntu14.04 amd64 [installed,automatic]

root@cc66c8fd0fda:/# ls -la /usr/lib/x86_64-linux-gnu/libmysql*
-rw-r--r-- 1 root root 5902058 Jul 12 13:37 /usr/lib/x86_64-linux-gnu/libmysqlclient.a
lrwxrwxrwx 1 root root      20 Jul 12 13:37 /usr/lib/x86_64-linux-gnu/libmysqlclient.so -> libmysqlclient.so.20
lrwxrwxrwx 1 root root      24 Jan 25  2016 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
-rw-r--r-- 1 root root 3355232 Jan 25  2016 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
lrwxrwxrwx 1 root root      24 Jul 12 13:37 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 -> libmysqlclient.so.20.3.1
-rw-r--r-- 1 root root 4283848 Jul 12 13:37 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.1
lrwxrwxrwx 1 root root      20 Jan 25  2016 /usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18 -> libmysqlclient.so.18
lrwxrwxrwx 1 root root      24 Jan 25  2016 /usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18.0.0 -> libmysqlclient.so.18.0.0
-rw-r--r-- 1 root root   18396 Jul 12 13:37 /usr/lib/x86_64-linux-gnu/libmysqlservices.a

[edmorley]

Ah yes - whilst mysql.com's libmysqlclient packages have strict mysql-common dependencies (ie: mysql-common (= 5.7.14-1ubuntu14.04)), the distro libmysqlclient18 package does not (mysql-common (>= 5.5.50-0ubuntu0.14.04.1)). I've spent the last few days dealing with only the mysql.com packages, and had run up against the strict dependency a few times, and had presumed the distro release would be similar.

I think we've found the first way in which the distro packages are not inferior :-)

I've updated the PR.

[edmorley]

Any news on this / https://help.heroku.com/tickets/376633 ?
I can't comment on that ticket any more, since it's been closed.

Thanks!

[dzuelke]

Haven't forgotten about it @edmorley! I can't share the exact details here, but expect it to land the week after next.

[edmorley]

Many thanks!

[edmorley]

Rebased on master.

Any news on this? Whilst the Heroku 16 stack is now in alpha, it's not yet suitable for production workloads, and until Travis support an Ubuntu 16.04 based environment (which is not happening for some time, judging by travis-ci/travis-ci#5695 (comment)), it will be hard to switch without sacrificing test-prod environment consistency.

[dzuelke]

Still investigating.

The problem is that the libmysqlclient20 update alone doesn't fix anything if the libraries don't also set a suitable value for the MYSQL_OPT_SSL_MODE option. Most of the libraries that people commonly use (https://github.com/PyMySQL/mysqlclient-python, https://github.com/python-oursql/oursql, https://github.com/datamapper/do) still only do mysql_ssl_set, which means the encrypted state is still strippable (see note at bottom of https://dev.mysql.com/doc/refman/5.7/en/mysql-ssl-set.html).

The only library that uses the new APIs and options (https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-11.html) is https://github.com/brianmario/mysql2 for Ruby; proper 5.7.11+ implementation here: brianmario/mysql2#793 (and 5.7.3-5.7.10 compatibility added afterwards: brianmario/mysql2@1363f4b)

[edmorley]

Ah but my understanding is that for mysql 5.7.x (libmysqlclient20) the SSL mode defaults to a correct value if ssl-ca was passed (which everyone has to do when connecting to RDS), meaning clients consuming libmysqlclient don't need to set the ssl mode explicitly.

Unfortunately the MySQL C API docs aren't clear about this:
https://dev.mysql.com/doc/refman/5.7/en/mysql-options.html

...and it's only mentioned in reference to the CLI options:
"Use of the --ssl-ca or --ssl-capath option implies --ssl-mode=VERIFY_CA, if --ssl-mode is not explicitly set otherwise."
(https://dev.mysql.com/doc/refman/5.7/en/secure-connection-options.html#option_general_ssl-mode)
...which I conceed isn't proof on its own, given the behaviour sometimes differs between CLI and C API for some options/defaults.

That said, the test we added after working around this (by manually vendoring the mysql 5.7.x libmysqlclient) shows that SSL cannot be stripped, unless there's a mistake in the test I've overlooked?
https://github.com/mozilla/treeherder/blob/796931e7b40f67d33ae4e34453889834550d67bc/tests/test_setup.py#L10-L35

(The test runs against a localhost mysql 5.6 instance that doesn't use SSL)

[dzuelke]

@johannes or @morgo can you shed some light on the above?

[morgo]

Sorry; I'm not strong in this area - I've passed the question along to a colleague, and I'll have to wait and see their response.

[dzuelke]

Thank you @morgo!

[morgo]

So a suggestion my colleague had, was that there is a viable workaround to enforce transport security on older broken clients on the server side (using a MySQL 5.7 server).

The option is to CREATE USER .. REQUIRE SSL.

[edmorley]

That unfortunately doesn't help, since it only enforces SSL on the server, not on the client. The CREATE USER .. REQUIRE SSL feature exists under mysql 5.6 too and doesn't prevent CVE-2015-3152.

ie: A malicious third party can strip SSL during connection, causing the client to send the credentials in plaintext, regardless of what the server accepts (the server has no control over what the client chooses to broadcast to the world in plaintext).

The malicious third party can even proxy the connection to the server using SSL for the second segment, so neither the client nor the server is any the wiser (the client's connection succeeds, and the server sees a connection using SSL). Such proxying might sound hypothetical, but there's a working example that does this already (https://github.com/duo-labs/mysslstrip).

This TLS stripping approach is similar to that which can be used against HTTP, which also has to be solved on the client (using eg HSTS). If you haven't seen the videos about sslstrip (eg https://vimeo.com/50018478#t=23m15s) they are pretty interesting.

[morgo]

Makes sense. Again, sorry not my area :)

[edmorley]

Any news on this? It's now coming up to 8 months since the PR was opened - it would be good to make an explicit fix/wontfix decision soon rather than leave this open. The compatibility risks don't appear to be changing over time, so either they are acceptable now or won't ever be.

Thanks!

[edmorley]

We've now migrated to heroku-16 which makes use of the stock Ubuntu libmysqlclient 5.7.x package, so this no longer affects us personally. We're also coming up to a year of this PR being open with no progress, and it's cluttering up my GitHub open PRs dashboard, so I'm just going to close it out.