9c568c58cf
As part of the new Discord <-> Status Community Import functionality, we're adding an API that extracts all discord categories and channels from a previously exported discord export file. These APIs can be used in clients to show the user what categories and channels will be imported later on. There are two APIs: 1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport []string) (*MessengerResponse, map[string]*discord.ImportError)` This takes a list of exported discord export (JSON) files (typically one per channel), reads them, and extracts the categories and channels into dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`) It also returns the oldest message timestamp found in all extracted channels. The API is synchronous and returns the extracted data as a `*MessengerResponse`. This allows to make the API available status-go's RPC interface. The error case is a `map[string]*discord.ImportError` where each key is a file path of a JSON file that we tried to extract data from, and the value a `discord.ImportError` which holds an error message and an error code, allowing for distinguishing between "critical" errors and "non-critical" errors. 2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport []string)` This is the asynchronous counterpart to `ExtractDiscordCategoriesAndChannels`. The reason this API has been added is because discord servers can have a lot of message and channel data, which causes `ExtractDiscordCategoriesAndChannels` to block the thread for too long, making apps potentially feel like they are stuck. This API runs inside a go routine, eventually calls `ExtractDiscordCategoriesAndChannels`, and then emits a newly introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients can react to. Failure of extraction has to be determined by the `discord.ImportErrors` emitted by the signal. **A note about exported discord history files** We expect users to export their discord histories via the [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild) tool. The tool allows to export the data in different formats, such as JSON, HTML and CSV. We expect users to have their data exported as JSON. Closes: https://github.com/status-im/status-desktop/issues/6690 |
||
|---|---|---|
| .. | ||
| migrations | ||
| README.md | ||
| cleaner.go | ||
| cleaner_test.go | ||
| db_key.go | ||
| db_key_test.go | ||
| limiter.go | ||
| limiter_test.go | ||
| mailserver.go | ||
| mailserver_db.go | ||
| mailserver_db_leveldb.go | ||
| mailserver_db_leveldb_test.go | ||
| mailserver_db_postgres.go | ||
| mailserver_db_postgres_test.go | ||
| mailserver_test.go | ||
| metrics.go | ||
| request.go | ||
README.md
MailServer
This document is meant to collect various information about our MailServer implementation.
Syncing between mail servers
It might happen that one mail server is behind other due to various reasons like a machine being down for a few minutes etc.
There is an option to fix such a mail server:
- SSH to a machine where this broken mail server runs,
- Add a mail server from which you want to sync:
# sudo might be not needed in your setup
$ echo '{"jsonrpc":"2.0","method":"admin_addPeer", "params": ["enode://c42f368a23fa98ee546fd247220759062323249ef657d26d357a777443aec04db1b29a3a22ef3e7c548e18493ddaf51a31b0aed6079bd6ebe5ae838fcfaf3a49@206.189.243.162:30504"], "id":1}' | \
sudo socat -d -d - UNIX-CONNECT:/docker/statusd-mail/data/geth.ipc
- Mark it as a trusted peer:
# sudo might be not needed in your setup
$ echo '{"jsonrpc":"2.0","method":"shh_markTrustedPeer", "params": ["enode://c42f368a23fa98ee546fd247220759062323249ef657d26d357a777443aec04db1b29a3a22ef3e7c548e18493ddaf51a31b0aed6079bd6ebe5ae838fcfaf3a49@206.189.243.162:30504"], "id":1}' | \
sudo socat -d -d - UNIX-CONNECT:/docker/statusd-mail/data/geth.ipc
- Finally, trigger the sync command:
# sudo might be not needed in your setup
$ echo '{"jsonrpc":"2.0","method":"shhext_syncMessages","params":[{"mailServerPeer":"enode://c42f368a23fa98ee546fd247220759062323249ef657d26d357a777443aec04db1b29a3a22ef3e7c548e18493ddaf51a31b0aed6079bd6ebe5ae838fcfaf3a49@206.189.243.162:30504", "to": 1550479953, "from": 1550393583, "limit": 1000}],"id":1}' | \
sudo socat -d -d - UNIX-CONNECT:/docker/statusd-mail/data/geth.ipc
You can add "followCursor": true if you want it to automatically download messages until the cursor is empty meaning all data was synced.
Debugging
To verify that your mail server received any responses, watch logs and seek for logs like this:
INFO [02-18|09:08:54.257] received sync response count=217 final=false err= cursor=[]
And it should finish with:
INFO [02-18|09:08:54.431] received sync response count=0 final=true err= cursor=[]