Distributed James Server — Operator guide

A toolbox is available to help an administrator diagnose issues:

Currently, an administrator can monitor mail processing failure through ERROR log review. We also recommend watching in Kibana INFO logs using the org.apache.james.transport.mailets.ToProcessor value as their logger. Metrics about mail repository size, and the corresponding Grafana boards are yet to be contributed.

Furthermore, given the default mailet container configuration, we recommend monitoring cassandra://var/mail/error/ to be empty.

WebAdmin exposes all utilities for reprocessing all mails in a mail repository or reprocessing a single mail in a mail repository.

In order to prevent unbounded processing that could consume unbounded resources. We can provide a CRON with limit parameter. Ex: 10 reprocessed per minute Note that it only support the reprocessing all mails.

Also, one can decide to delete all the mails of a mail repository or delete a single mail of a mail repository.

Performance of mail processing can be monitored via the mailet grafana board and matcher grafana board.

It is possible for the administrator of James to define the mailbox listeners he wants to use, by adding them in the listeners.xml configuration file. It’s possible also to add your own custom mailbox listeners. This enables to enhance capabilities of James as a Mail Delivery Agent. You can get more information about those here.

Currently, an administrator can monitor listeners failures through ERROR log review. Metrics regarding mailbox listeners can be monitored via mailbox_listeners grafana board and mailbox_listeners_rate grafana board.

Upon exceptions, a bounded number of retries are performed (with exponential backoff delays). If after those retries the listener is still failing to perform its operation, then the event will be stored in the Event Dead Letter. This API allows diagnosing issues, as well as redelivering the events.

To check that you have undelivered events in your system, you can first run the associated with event dead letter health check . You can explore Event DeadLetter content through WebAdmin. For this, list mailbox listener groups you will get a list of groups back, allowing you to check if those contain registered events in each by listing their failed events.

If you get failed events IDs back, you can as well check their details.

An easy way to solve this is just to trigger then the redeliver all events task. It will start reprocessing all the failed events registered in event dead letters.

In order to prevent unbounded processing that could consume unbounded resources. We can provide a CRON with limit parameter. Ex: 10 redelivery per minute

If for some other reason you don’t need to redeliver all events, you have more fine-grained operations allowing you to redeliver group events or even just redeliver a single event.

A projection of messages is maintained in OpenSearch via a listener plugged into the mailbox event bus in order to enable search features.

You can find more information about OpenSearch configuration here.

We recommend the administrator to run it in cron job to save storage volume.

Cassandra backend uses data duplication to workaround Cassandra query limitations. However, Cassandra is not doing transaction when writing in several tables, this can lead to consistency issues for a given piece of data. The consequence could be that the data is in a transient state (that should never appear outside of the system).

Because of the lack of transactions, it’s hard to prevent these kind of issues. We had developed some features to fix some existing cassandra inconsistency issues that had been reported to James.

When a Cassandra cluster is serving more than a James cluster, the keyspaces need isolation. It can be achieved by configuring James server with credentials preventing access or modification of other keyspaces.

We recommend you to not use the initial admin user of Cassandra and provide a different one with a subset of permissions for each application.

While Distributed James is shipped with default table configuration options, these settings should be refined depending of your usage.

These options are:

The compaction algorithms allow a tradeoff between background IO upon writes and reads. We recommend:

Bloom filters help avoiding unnecessary reads on SSTables. This probabilistic data structure can tell an entry absence from a SSTable, as well as the presence of an entry with an associated probability. If a lot of false positives are noticed, the size of the bloom filters can be increased.

As explained in this post, chunk size used upon compression allows a tradeoff between reads and writes. A smaller size will mean decreasing compression, thus it increases data being stored on disk, but allow lower chunks to be read to access data, and will favor reads. A bigger size will mean better compression, thus writing less, but it might imply reading bigger chunks.

Cassandra enables a key cache and a row cache. Key cache enables to skip reading the partition index upon reads, thus performing 1 read to the disk instead of 2. Enabling this cache is globally advised. Row cache stores the entire row in memory. It can be seen as an optimization, but it might actually use memory no longer available for instance for file system cache. We recommend turning it off on modern SSD hardware.

A review of your usage can be conducted using nodetool utility. For example nodetool tablestats {keyspace} allows reviewing the number of SSTables, the read/write ratios, bloom filter efficiency. nodetool tablehistograms {keyspace}.{table} might give insight about read/write performance.

Table level options can be changed using ALTER TABLE for example with the cqlsh utility. A full compaction might be needed in order for the changes to be taken into account.

A schema version indicates you which schema your James server is relying on. The schema version number tracks if a migration is required. For instance, when the latest schema version is 2, and the current schema version is 1, you might think that you still have data in the deprecated Message table in the database. Hence, you need to migrate these messages into the MessageV2 table. Once done, you can safely bump the current schema version to 2.

Relying on outdated schema version prevents you to benefit from the newest performance and safety improvements. Otherwise, there’s something very unexpected in the way we manage cassandra schema: we create new tables without asking the admin about it. That means your James version is always using the last tables but may also take into account the old ones if the migration is not done yet.