Support UNLOGGED tables

[citus-github-bot]

Issue by ergo70
Wednesday Jul 22, 2015 at 09:59 GMT
Originally opened as citusdata/pg_shard#126

---

Could master_create_worker_shards() be extended in order to optionally create UNLOGGED shard placements?

It seems to work when I change them manually but this is tedious. 😄 But it makes a big difference in write speed. Since there are replicas, the risk of UNLOGGED seems to be lower than in a single node database.

[citus-github-bot]

Comment by jasonmp85
Monday Jul 27, 2015 at 21:55 GMT

---

@ergo70: Are you certain you cannot achieve your performance goals through other means? We've seen pretty high insert rates without UNLOGGED by appropriately tuning PostgreSQL settings. I believe UNLOGGED tables are entirely truncated after a server crash, so your data loss would be total without backups (i.e. you'd lose all data in the table, not just data since some point in time).

One bad effect could occur if you execute a SELECT statement that somehow crashes a worker entirely (say, by executing buggy/unsafe extension code). pg_shard would realize the first replica failed to return a result, so it would continue to the second replica. If the bad code is written such that this query always crashes the server, the replica will crash too. You'd lose both replicas of your data just via that read operation. Presumably all of the other shards for this table would also be UNLOGGED, so any other shards on those two workers would also be truncated. This is quite an effect for a SELECT. I'm not saying none of our users are sophisticated enough to be able to responsibly make this choice, but I do think it might need some special treatment, or at least warnings.

The other side of the coin is that this is a PostgreSQL feature and so it's confusing to not support it. All my concerns above would also exist in a single-node setup and PostgreSQL has happily provided UNLOGGED for some time.

What precisely are your write throughput needs (rows/second)? I should be possible to hit them without UNLOGGED, but if that's absolutely impossible we may need to think outside the box.

[citus-github-bot]

Comment by ozgune
Tuesday Jul 28, 2015 at 05:25 GMT

---

I'm interested in this feature as well. That way, we could have replicated "in-memory tables" for Postgres. :)

[ozgune]

This issue relates to #54 and #184.

One thing to consider when adding UNLOGGED table support is that these tables don't benefit from streaming replication. This property relates to the masterless discussion as defined in #17 and #24.