This document describes a recommended set of database tables and algorithms for efficiently implementing JMAP. It is intended to serve as suggestions only; there may well be better ways to do it. The spec is the authoritative guide on what constitutes a conformant JMAP implementation.
A good way of assigning a message id is to use a secure hash function on the RFC5322 message. If you want to support having identical messages with different ids, you'll then need to keep hashing the result if there are collisions until you find an unused id. In general, I recommend not doing this.
A modification sequence, or modseq, is a 64-bit unsigned monotonically incrementing counter. Each user has their own modseq counter (in IMAP it's originally per-mailbox, but per user is backwards compatible with this). Every time a change occurs to data within the user, the modseq is incremented by one and the new value is associated with the changes. This is used in a number of data structures and algorithms below to efficiently calculate changes.
As ever in programming, get your data structures right and the server will practically write itself.
The mutable state and fields that need to be fetched for message list display for a message. The raw message itself is presumed to be stored in a separate (probably much slower) blob store, accessible again by the message id (or if ids are assigned by the blob store, add this as an extra property to the table).
- id:
String(The message id)
Properties below have the values as specified in the Message object in the JMAP Mail spec.
- blobId:
String(if different to message id) - threadId:
String - mailboxIds:
String[](Mutable) - keywords:
String[Boolean](Mutable) - from:
Emailer[]|null - to:
Emailer[]|null - subject:
String - date:
Date - size:
Number - preview:
String - attachments:
Attachment[]|null
Data sync properties:
- createdModSeq:
NumberThe modseq when the message was created. - updatedModSeq:
NumberThe modseq when the message was last modified. - deleted:
Date|nullThe timestamp when the message was deleted.
-
id:
String(The thread id; UUID suggested) -
subjectHash:
Byte[]Secure hash of the subject of the first message used to create the thread (after stripping fwd/re etc.). This is used for checking whether we need to split out a new thread when a message arrives as the subject has changed (see data structure 7). -
messages:
ThreadMessage[]The list of messages belonging to the thread, sorted with drafts immediately after the message they are in reply to, then by date order.A ThreadMessage has the following properties:
- id:
StringThe message id - mailboxIds:
String[] - isUnread:
Boolean - isFlagged:
Boolean
- id:
Data sync properties:
- createdModSeq:
NumberThe modseq when the thread was created. - updatedModSeq:
NumberThe modseq when the thread was last modified. - deleted:
Date|nullThe timestamp when the thread was deleted.
-
id:
String(The mailbox id; UUID suggested) -
name:
String -
parentId:
String|null -
role:
String|null -
sortOrder:
Number -
mayReadItems:
Boolean -
mayAddItems:
Boolean -
mayRemoveItems:
Boolean -
mayCreateChild:
Boolean -
mayRename:
Boolean -
mayDelete:
Boolean -
totalMessages:
Number -
unreadMessages:
Number -
totalThreads:
Number -
unreadThreads:
Number
Data sync properties:
-
createdModSeq:
NumberThe modseq when the mailbox was created. -
updatedModSeq:
NumberThe modseq when the mailbox was last modified. -
updatedNotCountsModSeq:
NumberThe modseq when any property other than the message counts was last modified. -
deleted:
Date|nullThe timestamp when the mailbox was deleted. -
highestUID:
NumberThe highest uid so far assigned to a message in the mailbox message list. -
messagesHighestModSeq:
NumberThe highest mod seq of any message in the mailbox. -
messageListLowModSeq:
NumberThe lowest modseq we can calculate message list updates from.
This data structure allows for IMAP support, and for very fast response to the most common message list fetch/updates queries (where the filter is a single mailbox, and sort is date descending).
-
id:
mailboxId . (Max_Int64 - MessageDate) . uidThe id is a concatenation of the mailbox id, then the message date inverted so date descending is the forward order, then the UID assigned. -
messageId:
StringThe message id. -
threadId:
StringThe thread id of the message. -
updatedModSeq:
StringThe currentupdatedModSeqof the message. -
created:
DateDate this message was added to the mailbox. -
deleted:
Date|nullDate this message was removed from the mailbox.
Note, the updatedModSeq is set at the same time as the deleted field when the message is removed from the mailbox, but not then updated even if the message is changed again. (If the message is added back to the mailbox it will be assigned a new uid, so a new record will be created).
A log of modseqs when messages are created/updated/deleted, for faster responses to getMessageUpdates.
-
id: The modseq of the change
-
changed:
String[]The list of message ids for messages that were changed or updated. -
removed:
String[]The list of message ids for messages that were deleted.
A log of modseqs when threads are created/updated/deleted, for faster responses to getThreadUpdates.
-
id: The modseq of the change
-
changed:
String[]The list of thread ids for threads that were changed or updated. -
removed:
String[]The list of thread ids for threads that were deleted.
This is solely used to look up the conversation to assign to a message on creation/import/delivery. Maps the RFC5322 message ids found in the the Message-ID, References and In-Reply-To headers of each message received to the thread id assigned to that message.
-
id: A secure hash of the RFC5322 message id. This may well be the message id if you assign message ids as recommended.
-
threadId:
StringThe thread id assigned to this message. -
lastSeen:
DateThe date of the last time this id was seen in a new message. This is used to clean up older entries after a while.
The suggested rule for connecting messages into threads is this:
If two messages share a common RFC5322 Message Id in the set of such ids within of each message, and the messages have the same Subject header hash (after stripping any preceding Re:/Fwd: and trimming white space from either end), then they should belong in the same thread. Otherwise they should belong in different threads.
These are singleton values that need to be stored. If using a DB table, would be id/number values.
- highModSeq:
NumberThe highest mod seq that has been assigned, across all types. - highModSeqMessage:
NumberThe highest mod seq assigned to a message. - highModSeqThread:
NumberThe highest mod seq assigned to a thread. - highModSeqMailbox:
NumberThe highest mod seq assigned to a mailbox. - lowModSeqMessage:
NumberThe lowest mod seq from which we can calculate updates. Periodically we shorten the MessageChangeLog by removing entries from the beginning. At this point, the highest modseq value removed is set as thelowModSeqMessage. - lowModSeqThread:
NumberThe lowest mod seq from which we can calculate updates. Periodically we shorten the ThreadChangeLog by removing entries from the beginning. At this point, the highest modseq value removed is set as thelowModSeqThread. - lowModSeqMailbox:
NumberThe lowest mod seq from which we can calculate updates. Periodically we completely delete mailboxes that were marked deleted. At this point, if higher than the current value, theupdatedModSeqof the mailbox is set as the newlowModSeqMailbox.
There are two bits of data not included in the main database.
The set of full raw messages, possibly in a blob store.
For searching messages at any reasonable speed, an index is required from textual content within the message to the message id. Describing how this should work is beyond the scope of this guide.
The state property to return with getter calls to each type is:
-
Message: The
highModSeqMessagevalue. -
Thread: The
highModSeqThreadvalue. -
Mailbox: The
highModSeqMailboxvalue. -
MessageList: The state property to use depends on the sort/filter of the message list, as some optimisations can be made for common cases. The state string should encode one or two values. In all cases, you need to include a modseq value, which should be on of the following:
- If the sort or filter includes a mutable thread property
(i.e threadUnread or threadFlagged), use
max(highModSeqMessage, highModSeqThread). - Otherwise, if the filter is or can be transformed to:
AND( inMailbox: mailboxId, <other conditions>), useMailbox.messagesHighestModSeq - Otherwise, use
highModSeqMessage
If the filter is, or can be transformed to:
AND( inMailbox: mailboxId, <other conditions>), also encodeMailbox.highestUIDinto the state string; see getMessageListUpdates algorithm below for how these values will be used. - If the sort or filter includes a mutable thread property
(i.e threadUnread or threadFlagged), use
If the modseq given == the current highModSeqMessage value, there are no changes. If it is lower than lowModSeqMessage we cannot calculate changes.
If there are changes, find the first entry in the MessageChangeLog with an id >= the given modseq, and read forward from there.
If the modseq given == the current highModSeqThread value, there are no changes. If it is lower than lowModSeqThread we cannot calculate changes.
If there are changes, find the first entry in the ThreadChangeLog with an id >= the given modseq, and read forward from there.
If the modseq given == the current highModSeqMailbox value, there are no changes. If it is lower than lowModSeqMailbox we cannot calculate changes.
If there are changes, iterate through the set of Mailboxes comparing their updatedModSeq to the client mod seq. If higher, the mailbox has changed (if deleted is not null it has been deleted, otherwise it was created or updated). If the updatedNotCountsModSeq is not higher, only counts have changed.
Check the current state string as would be returned for getMessageList with the same arguments. If it is the same as the state given, nothing has changed so you can return a response immediately. Otherwise…
First you need to get the complete list of messages that match the given filter. In the common case of…
filter == { inMailbox: <messageId> }
…you are simply fetching the list of messages in a mailbox. This list is already pre-calculated for each mailbox and kept on disk (data structure 4). You can simply slurp this into memory and sort if needed (again, in the common case of sorting date-descending, it will be pre-sorted). Skip any messages marked deleted.
If the filter is more complex, you will need to do more work to get the set of matching messages and sort it. If there is a String component to the filter, first use the message index to get a set of matches. If there is a single inMailbox component, use the MailboxMessageList structure to get the list of messages in that mailbox. Finally iterate through and lookup each potential match in the messages table (data structure 1) to apply any other components of the filter. Finally sort as specified.
Once you have the complete message list, you can find the requested section to return to the client. Since a client is likely to fetch a different portion of the same message list soon after, it is beneficial if the server can keep the last list requested by the user in a cache for a short time.
let collapseThreads = args.collapseThreads
let position = args.position
let anchor = args.anchor
let anchorOffset = args.anchorOffset
let limit = args.limit
let total = 0
let messageIds = [] # NB Max size of array is limit
let threadIds = [] # NB Max size of array is limit
# If not collapsing threads, we can just jump to the required section
if !collapseThreads {
total = messageList.length
for i = position; i < total; i = i + 1 {
messageIds.push( msg.id )
threadIds.push( msg.threadId )
}
} else {
# Optimisation for the common case
let totalIsKnown = filter is just mailbox
let SeenThread = new Set()
let numFound = 0
foreach msg in sortedFilteredList {
if !SeenThread{ msg.threadId } {
SeenThread.add( msg.threadId )
total += 1
if position >= total && numFound < limit {
messageIds.push( msg.id )
threadIds.push( msg.threadId )
numFound += 1
if numFound == limit && totalIsKnown {
break;
}
}
}
}
if totalIsKnown {
total = mailbox.totalThreads
}
}
For the common case of…
filter == { inMailbox: <messageId> }
… take the complete message list for the mailbox (data structure 4), including those marked deleted. Sort if necessary. The state given by the user (as returned from getMessageList) contains the previous highestUID and a highestModSeq. Then:
let index = -1
let total = 0
let added = []
let removed = []
let collapseThreads = args.collapseThreads
let uptoHasBeenFound = false
# A mutable sort is one which sorts by a mutable property, e.g.
# sort flagged messages/threads first.
let isMutable = sort.isMutable()
# Does the sort include a thread property (thread unread/thread flagged)
# or just message properties?
let isOnThreadUnreadOrFlagged = sort.isOnThreadUnreadOrFlagged()
# An exemplar is the first message in each thread in the list, given the
# sort order that
# The old exemplar is the exemplar in the client's old state.
let SeenExemplar = collapseThreads ? new Set() : null
let SeenOldExemplar = collapseThreads ? new Set() : null
foreach listMsg in messageList {
let isNewExemplar = false
let isOldExemplar = false
let isNew = ( listMsg.uid > args.highestUID )
let isChanged = ( listMsg.updatedModSeq > args.highestModSeq )
let isDeleted = ( listMsg.deleted != null )
let wasDeleted = ( isDeleted && !isChanged )
# Is this message the current exemplar?
if !isDeleted &&
( !collapseThreads || !SeenExemplar{ listMsg.threadId } ) {
isNewExemplar = true
index += 1
total += 1
if collapseThreads {
SeenExemplar.set( listMsg.threadId )
}
}
# Was this message an old exemplar?
# 1. Must not have been added to mailbox after the client's state
# 2. Must have been removed from mailbox before the client's state
# 3. Must not have already found the old exemplar
if !isNew && !wasDeleted &&
( !collapseThreads || !SeenOldExemplar{ listMsg.threadId } ) {
isOldExemplar = true
if collapseThreads {
SeenOldExemplar.set( listMsg.threadId )
}
}
if isOldExemplar && !isNewExemplar {
removed.push({
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
}
else if !isOldExemplar && isNewExemplar {
added.push({
index: index,
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
}
# Special case for mutable sorts (based on isFlagged/isUnread)
if isMutable && isOldExemplar && isNewExemplar {
# Has the isUnread/isFlagged status of the message/thread
# (as appropriate) possibly changed since the client's state?
let modSeq = isOnThreadUnreadOrFlagged ?
getThread( listMsg.threadId ).updatedModSeq :
listMsg.updatedModSeq
# If so, we need to remove the exemplar from the client view and add
# it back in at the correct position.
if modSeq > args.modSeq {
removed.push({
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
added.push({
index: index,
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
}
}
# If this is the last message the client cares about, we can stop here
# and just return what we've calculated so far. We already know the total
# count for this message list as we keep it pre calculated and cached in
# the Mailbox object.
#
# However, if the sort is mutable we can't break early, as messages may
# have moved from the region we care about to lower down the list.
if !isMutable && !isNew && listMsg.messageId == args.upto {
break
}
} # End loop
total = getTotal( mailbox, collapseThreads )
For other filters, you can use the following algorithm instead, but it does require a complete scan of all messages:
let index = -1
let total = 0
let added = []
let removed = []
let collapseThreads = args.collapseThreads
let uptoHasBeenFound = false
# A mutable filter/sort is one which uses a mutable property, e.g.
# flagged/unread state
let isMutable = sort.isMutable() || filter.isMutable()
# Does the sort or filter include a thread property (thread unread/thread
# flagged) or just message properties?
let isOnThreadUnreadOrFlagged =
sort.isOnThreadUnreadOrFlagged() || filter.isOnThreadUnreadOrFlagged()
# An exemplar is the first message in each thread in the list, given the
# sort order that
# The old exemplar is the exemplar in the client's old state.
let SeenExemplar = collapseThreads ? new Set() : null
let SeenOldExemplar = collapseThreads ? new Set() : null
# Get the full list of messages. Any non-mutable filter components
# may be applied at this stage, if possible.
let messages = allMessages().sortBy( args.sort )
foreach message in messages {
let isNewExemplar = false
let isOldExemplar = false
let isMatch = filter.matches( message )
let isNew = ( message.createdModSeq > args.highestModSeq )
let isChanged = ( message.updatedModSeq > args.highestModSeq )
let isDeleted = ( message.deleted != null )
let wasDeleted = ( isDeleted && !isChanged )
# Check for thread changed if necessary
if isOnThreadUnreadOrFlagged && !isChanged &&
getThread( listMsg.threadId ).updatedModSeq > args.highestModSeq {
isChanged = true
}
# Is this message the current exemplar?
if isMatch && !isDeleted &&
( !collapseThreads || !SeenExemplar{ message.threadId } ) {
isNewExemplar = true
index += 1
total += 1
if collapseThreads {
SeenExemplar.set( listMsg.threadId )
}
}
# Was this message an old exemplar?
# 1. Must not have been created after the client's state
# 2. Must have been deleted before the client's state
# 3. Must match the filter, or have changed and the sort/filter is on a
mutable property so it may have matched before.
# 4. Must not have already found the old exemplar, or filter/sort is on a
mutable property.
if !isNew && !wasDeleted &&
( isMatch || ( isMutable && isChanged ) ) &&
( !collapseThreads || isMutable ||
!SeenOldExemplar{ listMsg.threadId } ) {
isOldExemplar = true
if collapseThreads && !isMutable {
SeenOldExemplar.set( listMsg.threadId )
}
}
if isOldExemplar && ( !isNewExemplar || isMutable ) {
removed.push({
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
}
if isNewExemplar && ( !isOldExemplar || isMutable ) {
added.push({
index: index,
messageId: listMsg.messageId,
threadId: listMsg.threadId
})
}
} # End loop
As mentioned in RFC-5230 Sieve Vacation extension, implementations should avoid sending vacation responses to mailing lists. Implementations should also avoid sending responses to well-known addresses like "MAILER-DAEMON", "LISTSERV", "majordomo", and other addresses typically used only by automated systems. Additionally, addresses ending in "-request" or begining in "owner-", i.e., reserved for mailing list software, should also not receive vacation responses. Implementations should not respond to any message that contains a "List-Id" RFC-2919, "List-Help", "List-Subscribe", "List-Unsubscribe", "List-Post", "List-Owner", or "List-Archive" RFC-2369 header field.
Also, be careful about infinite loops. Vacation responses should not be sent in response to another vacation response. To avoid doing so, as specified in RFC-5230, an Auto-Submitted header with a value of "auto-replied" should be included in any vacation message sent. Implementations should not send responses to email with an Auto-Submitted header with a value of "auto-replied".
The server must keep track of sent notifications, in order to avoid sending notifications twice to the same recipient during the duration of a given vacation. Implementers must take care that if the vacation is modified, previous tracking information should be discarded.
Finaly, implementations are encouraged to comply with RFC-3824, which defines a personal responder. To do so:
- The Date field should be set to the date and time when the vacation response was generated. Note that this may not be the same as the time the message was delivered to the user.
- The From field should be set to the address of the given account.
- The To field should be set to the address of the recipient of the response.
- An Auto-Submitted field with a value of "auto-replied" should be included in the message header of any vacation message sent.
- Replies must have the In-Reply-To field set to the Message-ID of the original message, and the References field should be updated with the Message-ID of the original message.
If the original message lacks a Message-ID, an In-Reply-To need not be generated, and References need not be changed. RFC-2822 section 3.6.4 provides a complete description of how References fields should be generated.
The upload handler is a standard HTTP resource, so must conform with the HTTP standard. For reference, here are some of the appropriate HTTP responses to return for client errors:
The request was malformed (this includes the case where an X-JMAP-AccountId header is sent with a value that does not exist).
The Authorization header was missing or did not contain a valid token. Reauthenticate and then retry the request. As per the HTTP spec, the response MUST have a WWW-Authenticate header listing the available authentication schemes.
The file is larger than the maximum size the server is willing to accept for a single file.
An unacceptable type is uploaded. The server MAY choose to not allow certain content types to be uploaded, such as executable files.
The client has made too many upload requests recently, or has too many concurrent uploads currently in progress. The response MAY include a Retry-After header indicating how long to wait before making a new request.