status-go/services/ext/signal.go

130 lines
4.8 KiB
Go
Raw Normal View History

package ext
import (
"github.com/status-im/status-go/eth-node/types"
"github.com/status-im/status-go/protocol"
"github.com/status-im/status-go/protocol/communities"
feat: introduce messenger APIs to extract discord channels As part of the new Discord <-> Status Community Import functionality, we're adding an API that extracts all discord categories and channels from a previously exported discord export file. These APIs can be used in clients to show the user what categories and channels will be imported later on. There are two APIs: 1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport []string) (*MessengerResponse, map[string]*discord.ImportError)` This takes a list of exported discord export (JSON) files (typically one per channel), reads them, and extracts the categories and channels into dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`) It also returns the oldest message timestamp found in all extracted channels. The API is synchronous and returns the extracted data as a `*MessengerResponse`. This allows to make the API available status-go's RPC interface. The error case is a `map[string]*discord.ImportError` where each key is a file path of a JSON file that we tried to extract data from, and the value a `discord.ImportError` which holds an error message and an error code, allowing for distinguishing between "critical" errors and "non-critical" errors. 2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport []string)` This is the asynchronous counterpart to `ExtractDiscordCategoriesAndChannels`. The reason this API has been added is because discord servers can have a lot of message and channel data, which causes `ExtractDiscordCategoriesAndChannels` to block the thread for too long, making apps potentially feel like they are stuck. This API runs inside a go routine, eventually calls `ExtractDiscordCategoriesAndChannels`, and then emits a newly introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients can react to. Failure of extraction has to be determined by the `discord.ImportErrors` emitted by the signal. **A note about exported discord history files** We expect users to export their discord histories via the [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild) tool. The tool allows to export the data in different formats, such as JSON, HTML and CSV. We expect users to have their data exported as JSON. Closes: https://github.com/status-im/status-desktop/issues/6690
2022-07-13 09:33:53 +00:00
"github.com/status-im/status-go/protocol/discord"
2018-05-03 07:35:58 +00:00
"github.com/status-im/status-go/signal"
)
// EnvelopeSignalHandler sends signals when envelope is sent or expired.
type EnvelopeSignalHandler struct{}
// EnvelopeSent triggered when envelope delivered atleast to 1 peer.
func (h EnvelopeSignalHandler) EnvelopeSent(identifiers [][]byte) {
signal.SendEnvelopeSent(identifiers)
}
// EnvelopeExpired triggered when envelope is expired but wasn't delivered to any peer.
func (h EnvelopeSignalHandler) EnvelopeExpired(identifiers [][]byte, err error) {
signal.SendEnvelopeExpired(identifiers, err)
}
mailserver pagination (#1039) * mailserver sends envelopes in descending order * add limit value in mailserver request payload * mailserver sends messages up to the limit specified in the request * update Archive method to return key and error * processRequest returns the next page cursor * add cursor to mailserver request * add limit and cursor to request payload * fix request limit encoding * wait for request completed event in TrackerSuite/TestRequestCompleted * add cursor to mailserver response * fix cursor position in payload * add e2e test for mail server pagination * validate mail server response size * remove old limitReached var * fix lint warnings * add whisper patch * fix tests after rebase * check all return values to avoid lint warnings * check that all messages have been retrieved after 2 paginated requests * fix lint warnings * rename geth patch * merge mailserver patches into one * add last envelope hash to mailserver response and EventEnvelopeAvailable event * update whisper patch * add docs to MailServerResponse * update whisper patch * fix tests and lint warnings * send mailserver response data on EventMailServerRequestCompleted signal * update tracker tests * optimise pagination test waiting for mailserver to archive only before requesting * rollback mailserver interface changes * refactoring and docs changes * fix payload size check to determine if a limit is specified * add more docs to the processRequest method * add constants for request payload field lengths * add const noLimits to specify that limit=0 means no limits
2018-07-02 07:38:10 +00:00
// MailServerRequestCompleted triggered when the mailserver sends a message to notify that the request has been completed
func (h EnvelopeSignalHandler) MailServerRequestCompleted(requestID types.Hash, lastEnvelopeHash types.Hash, cursor []byte, err error) {
signal.SendMailServerRequestCompleted(requestID, lastEnvelopeHash, cursor, err)
}
mailserver pagination (#1039) * mailserver sends envelopes in descending order * add limit value in mailserver request payload * mailserver sends messages up to the limit specified in the request * update Archive method to return key and error * processRequest returns the next page cursor * add cursor to mailserver request * add limit and cursor to request payload * fix request limit encoding * wait for request completed event in TrackerSuite/TestRequestCompleted * add cursor to mailserver response * fix cursor position in payload * add e2e test for mail server pagination * validate mail server response size * remove old limitReached var * fix lint warnings * add whisper patch * fix tests after rebase * check all return values to avoid lint warnings * check that all messages have been retrieved after 2 paginated requests * fix lint warnings * rename geth patch * merge mailserver patches into one * add last envelope hash to mailserver response and EventEnvelopeAvailable event * update whisper patch * add docs to MailServerResponse * update whisper patch * fix tests and lint warnings * send mailserver response data on EventMailServerRequestCompleted signal * update tracker tests * optimise pagination test waiting for mailserver to archive only before requesting * rollback mailserver interface changes * refactoring and docs changes * fix payload size check to determine if a limit is specified * add more docs to the processRequest method * add constants for request payload field lengths * add const noLimits to specify that limit=0 means no limits
2018-07-02 07:38:10 +00:00
// MailServerRequestExpired triggered when the mailserver request expires
func (h EnvelopeSignalHandler) MailServerRequestExpired(hash types.Hash) {
signal.SendMailServerRequestExpired(hash)
}
2019-07-01 09:39:51 +00:00
// PublisherSignalHandler sends signals on protocol events
type PublisherSignalHandler struct{}
func (h PublisherSignalHandler) DecryptMessageFailed(pubKey string) {
signal.SendDecryptMessageFailed(pubKey)
}
func (h PublisherSignalHandler) BundleAdded(identity string, installationID string) {
signal.SendBundleAdded(identity, installationID)
}
func (h PublisherSignalHandler) NewMessages(response *protocol.MessengerResponse) {
signal.SendNewMessages(response)
2019-07-01 10:00:46 +00:00
}
func (h PublisherSignalHandler) Stats(stats types.StatsSummary) {
signal.SendStats(stats)
}
// MessengerSignalHandler sends signals on messenger events
type MessengerSignalsHandler struct{}
// MessageDelivered passes information that message was delivered
func (m MessengerSignalsHandler) MessageDelivered(chatID string, messageID string) {
signal.SendMessageDelivered(chatID, messageID)
}
// BackupPerformed passes information that a backup was performed
func (m MessengerSignalsHandler) BackupPerformed(lastBackup uint64) {
signal.SendBackupPerformed(lastBackup)
}
// MessageDelivered passes info about community that was requested before
func (m MessengerSignalsHandler) CommunityInfoFound(community *communities.Community) {
signal.SendCommunityInfoFound(community)
}
2021-03-25 15:15:22 +00:00
func (m *MessengerSignalsHandler) MessengerResponse(response *protocol.MessengerResponse) {
PublisherSignalHandler{}.NewMessages(response)
}
func (m *MessengerSignalsHandler) HistoryRequestStarted(requestID string, numBatches int) {
signal.SendHistoricMessagesRequestStarted(requestID, numBatches)
}
func (m *MessengerSignalsHandler) HistoryRequestBatchProcessed(requestID string, batchIndex int, numBatches int) {
signal.SendHistoricMessagesRequestBatchProcessed(requestID, batchIndex, numBatches)
}
func (m *MessengerSignalsHandler) HistoryRequestFailed(requestID string, err error) {
signal.SendHistoricMessagesRequestFailed(requestID, err)
}
func (m *MessengerSignalsHandler) HistoryRequestCompleted(requestID string) {
signal.SendHistoricMessagesRequestCompleted(requestID)
}
func (m *MessengerSignalsHandler) HistoryArchivesProtocolEnabled() {
signal.SendHistoryArchivesProtocolEnabled()
}
func (m *MessengerSignalsHandler) HistoryArchivesProtocolDisabled() {
signal.SendHistoryArchivesProtocolDisabled()
}
func (m *MessengerSignalsHandler) CreatingHistoryArchives(communityID string) {
signal.SendCreatingHistoryArchives(communityID)
}
func (m *MessengerSignalsHandler) NoHistoryArchivesCreated(communityID string, from int, to int) {
signal.SendNoHistoryArchivesCreated(communityID, from, to)
}
func (m *MessengerSignalsHandler) HistoryArchivesCreated(communityID string, from int, to int) {
signal.SendHistoryArchivesCreated(communityID, from, to)
}
func (m *MessengerSignalsHandler) HistoryArchivesSeeding(communityID string) {
signal.SendHistoryArchivesSeeding(communityID)
}
func (m *MessengerSignalsHandler) HistoryArchivesUnseeded(communityID string) {
signal.SendHistoryArchivesUnseeded(communityID)
}
func (m *MessengerSignalsHandler) HistoryArchiveDownloaded(communityID string, from int, to int) {
signal.SendHistoryArchiveDownloaded(communityID, from, to)
}
func (m *MessengerSignalsHandler) StatusUpdatesTimedOut(statusUpdates *[]protocol.UserStatus) {
signal.SendStatusUpdatesTimedOut(statusUpdates)
}
feat: introduce messenger APIs to extract discord channels As part of the new Discord <-> Status Community Import functionality, we're adding an API that extracts all discord categories and channels from a previously exported discord export file. These APIs can be used in clients to show the user what categories and channels will be imported later on. There are two APIs: 1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport []string) (*MessengerResponse, map[string]*discord.ImportError)` This takes a list of exported discord export (JSON) files (typically one per channel), reads them, and extracts the categories and channels into dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`) It also returns the oldest message timestamp found in all extracted channels. The API is synchronous and returns the extracted data as a `*MessengerResponse`. This allows to make the API available status-go's RPC interface. The error case is a `map[string]*discord.ImportError` where each key is a file path of a JSON file that we tried to extract data from, and the value a `discord.ImportError` which holds an error message and an error code, allowing for distinguishing between "critical" errors and "non-critical" errors. 2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport []string)` This is the asynchronous counterpart to `ExtractDiscordCategoriesAndChannels`. The reason this API has been added is because discord servers can have a lot of message and channel data, which causes `ExtractDiscordCategoriesAndChannels` to block the thread for too long, making apps potentially feel like they are stuck. This API runs inside a go routine, eventually calls `ExtractDiscordCategoriesAndChannels`, and then emits a newly introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients can react to. Failure of extraction has to be determined by the `discord.ImportErrors` emitted by the signal. **A note about exported discord history files** We expect users to export their discord histories via the [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild) tool. The tool allows to export the data in different formats, such as JSON, HTML and CSV. We expect users to have their data exported as JSON. Closes: https://github.com/status-im/status-desktop/issues/6690
2022-07-13 09:33:53 +00:00
func (m *MessengerSignalsHandler) DiscordCategoriesAndChannelsExtracted(categories []*discord.Category, channels []*discord.Channel, oldestMessageTimestamp int64, errors map[string]*discord.ImportError) {
signal.SendDiscordCategoriesAndChannelsExtracted(categories, channels, oldestMessageTimestamp, errors)
}