status-go/protocol/v1/status_message.go
Samuel Hawksby-Robinson 07e46714f0
Anon Metrics Broadcast (#2198)
* Protobufs and adapters

* Added basic anon metric service and config init

* Added fibonacci interval incrementer

* Added basic Client.Start func and integrated interval incrementer

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Moved incrementer into dedicated file

* Resolve incrementer test fail

* Finalised the main loop functionality

* Implemented delete loop framework

* Updated adaptors file name

* Added delete loop delay and quit, and tweak on RawMessage gen

* Completed delete loop logic

* Added DBLock to prevent deletion during mainLoop

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Function name update

* Added sample config files for client and server

* Fixes and testing for low level e2e

* make generate

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* delinting

* Fixing database tests

* Attempted fix of does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work

* Lint for the lint god

* Why doesn't the linter list all its problems at once?

* test tweaks

* Fix for wakuV2 change

* DB reset change

* Fix for postgres db migrations fails

* More robust implementation of postgres test setup and teardown

* Added block for anon metrics functionality

* Version Bump to 0.84.0

* Added test to check anon metrics broadcast is deactivated

* Protobufs and adapters

* Added basic anon metric service and config init

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Added sample config files for client and server

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
2021-09-01 13:02:18 +01:00

277 lines
9.9 KiB
Go

package protocol
import (
"crypto/ecdsa"
"encoding/json"
"log"
"reflect"
"github.com/golang/protobuf/proto"
"github.com/jinzhu/copier"
"github.com/pkg/errors"
"github.com/status-im/status-go/eth-node/crypto"
"github.com/status-im/status-go/eth-node/types"
"github.com/status-im/status-go/protocol/encryption"
"github.com/status-im/status-go/protocol/encryption/multidevice"
"github.com/status-im/status-go/protocol/encryption/sharedsecret"
"github.com/status-im/status-go/protocol/protobuf"
)
type StatusMessageT int
// StatusMessage is any Status Protocol message.
type StatusMessage struct {
// TransportMessage is the parsed message received from the transport layer, i.e the input
TransportMessage *types.Message `json:"transportMessage"`
// Type is the type of application message contained
Type protobuf.ApplicationMetadataMessage_Type `json:"-"`
// ParsedMessage is the parsed message by the application layer, i.e the output
ParsedMessage *reflect.Value `json:"-"`
// TransportPayload is the payload as received from the transport layer
TransportPayload []byte `json:"-"`
// DecryptedPayload is the payload after having been processed by the encryption layer
DecryptedPayload []byte `json:"decryptedPayload"`
// UnwrappedPayload is the payload after having been unwrapped from the applicaition metadata layer
UnwrappedPayload []byte `json:"unwrappedPayload"`
// ID is the canonical ID of the message
ID types.HexBytes `json:"id"`
// Hash is the transport layer hash
Hash []byte `json:"-"`
// Dst is the targeted public key
Dst *ecdsa.PublicKey
// TransportLayerSigPubKey contains the public key provided by the transport layer
TransportLayerSigPubKey *ecdsa.PublicKey `json:"-"`
// ApplicationMetadataLayerPubKey contains the public key provided by the application metadata layer
ApplicationMetadataLayerSigPubKey *ecdsa.PublicKey `json:"-"`
// Installations is the new installations returned by the encryption layer
Installations []*multidevice.Installation
// SharedSecret is the shared secret returned by the encryption layer
SharedSecrets []*sharedsecret.Secret
}
// Temporary JSON marshaling for those messages that are not yet processed
// by the go code
func (m *StatusMessage) MarshalJSON() ([]byte, error) {
item := struct {
ID types.HexBytes `json:"id"`
Payload string `json:"payload"`
From types.HexBytes `json:"from"`
Timestamp uint32 `json:"timestamp"`
}{
ID: m.ID,
Payload: string(m.UnwrappedPayload),
Timestamp: m.TransportMessage.Timestamp,
From: m.TransportMessage.Sig,
}
return json.Marshal(item)
}
// SigPubKey returns the most important signature, from the application layer to transport
func (m *StatusMessage) SigPubKey() *ecdsa.PublicKey {
if m.ApplicationMetadataLayerSigPubKey != nil {
return m.ApplicationMetadataLayerSigPubKey
}
return m.TransportLayerSigPubKey
}
func (m *StatusMessage) Clone() (*StatusMessage, error) {
copy := &StatusMessage{}
err := copier.Copy(&copy, m)
return copy, err
}
func (m *StatusMessage) HandleTransport(shhMessage *types.Message) error {
publicKey, err := crypto.UnmarshalPubkey(shhMessage.Sig)
if err != nil {
return errors.Wrap(err, "failed to get signature")
}
m.TransportMessage = shhMessage
m.Hash = shhMessage.Hash
m.TransportLayerSigPubKey = publicKey
m.TransportPayload = shhMessage.Payload
if shhMessage.Dst != nil {
publicKey, err := crypto.UnmarshalPubkey(shhMessage.Dst)
if err != nil {
return err
}
m.Dst = publicKey
}
return nil
}
func (m *StatusMessage) HandleEncryption(myKey *ecdsa.PrivateKey, senderKey *ecdsa.PublicKey, enc *encryption.Protocol, skipNegotiation bool) error {
// As we handle non-encrypted messages, we make sure that DecryptPayload
// is set regardless of whether this step is successful
m.DecryptedPayload = m.TransportPayload
// Nothing to do
if skipNegotiation {
return nil
}
var protocolMessage encryption.ProtocolMessage
err := proto.Unmarshal(m.TransportPayload, &protocolMessage)
if err != nil {
return errors.Wrap(err, "failed to unmarshal ProtocolMessage")
}
response, err := enc.HandleMessage(
myKey,
senderKey,
&protocolMessage,
m.Hash,
)
if err != nil {
return errors.Wrap(err, "failed to handle Encryption message")
}
m.DecryptedPayload = response.DecryptedMessage
m.Installations = response.Installations
m.SharedSecrets = response.SharedSecrets
return nil
}
func (m *StatusMessage) HandleApplicationMetadata() error {
message, err := protobuf.Unmarshal(m.DecryptedPayload)
if err != nil {
return err
}
recoveredKey, err := message.RecoverKey()
if err != nil {
return err
}
m.ApplicationMetadataLayerSigPubKey = recoveredKey
// Calculate ID using the wrapped record
m.ID = MessageID(recoveredKey, m.DecryptedPayload)
m.UnwrappedPayload = message.Payload
m.Type = message.Type
return nil
}
func (m *StatusMessage) HandleApplication() error {
switch m.Type {
case protobuf.ApplicationMetadataMessage_CHAT_MESSAGE:
return m.unmarshalProtobufData(new(protobuf.ChatMessage))
case protobuf.ApplicationMetadataMessage_MEMBERSHIP_UPDATE_MESSAGE:
return m.unmarshalProtobufData(new(protobuf.MembershipUpdateMessage))
case protobuf.ApplicationMetadataMessage_ACCEPT_REQUEST_ADDRESS_FOR_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.AcceptRequestAddressForTransaction))
case protobuf.ApplicationMetadataMessage_SEND_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.SendTransaction))
case protobuf.ApplicationMetadataMessage_REQUEST_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.RequestTransaction))
case protobuf.ApplicationMetadataMessage_DECLINE_REQUEST_ADDRESS_FOR_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.DeclineRequestAddressForTransaction))
case protobuf.ApplicationMetadataMessage_DECLINE_REQUEST_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.DeclineRequestTransaction))
case protobuf.ApplicationMetadataMessage_REQUEST_ADDRESS_FOR_TRANSACTION:
return m.unmarshalProtobufData(new(protobuf.RequestAddressForTransaction))
case protobuf.ApplicationMetadataMessage_CONTACT_UPDATE:
return m.unmarshalProtobufData(new(protobuf.ContactUpdate))
case protobuf.ApplicationMetadataMessage_PIN_MESSAGE:
return m.unmarshalProtobufData(new(protobuf.PinMessage))
case protobuf.ApplicationMetadataMessage_SYNC_INSTALLATION:
return m.unmarshalProtobufData(new(protobuf.SyncInstallation))
case protobuf.ApplicationMetadataMessage_SYNC_INSTALLATION_CONTACT:
log.Printf("Sync installation contact")
return m.unmarshalProtobufData(new(protobuf.SyncInstallationContact))
case protobuf.ApplicationMetadataMessage_SYNC_INSTALLATION_PUBLIC_CHAT:
return m.unmarshalProtobufData(new(protobuf.SyncInstallationPublicChat))
case protobuf.ApplicationMetadataMessage_SYNC_INSTALLATION_ACCOUNT:
return m.unmarshalProtobufData(new(protobuf.SyncInstallationAccount))
case protobuf.ApplicationMetadataMessage_PAIR_INSTALLATION:
return m.unmarshalProtobufData(new(protobuf.PairInstallation))
case protobuf.ApplicationMetadataMessage_SYNC_INSTALLATION_COMMUNITY:
return m.unmarshalProtobufData(new(protobuf.SyncCommunity))
case protobuf.ApplicationMetadataMessage_CONTACT_CODE_ADVERTISEMENT:
return m.unmarshalProtobufData(new(protobuf.ContactCodeAdvertisement))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_REQUEST:
return m.unmarshalProtobufData(new(protobuf.PushNotificationRequest))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_REGISTRATION_RESPONSE:
return m.unmarshalProtobufData(new(protobuf.PushNotificationRegistrationResponse))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_QUERY:
return m.unmarshalProtobufData(new(protobuf.PushNotificationQuery))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_QUERY_RESPONSE:
return m.unmarshalProtobufData(new(protobuf.PushNotificationQueryResponse))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_RESPONSE:
return m.unmarshalProtobufData(new(protobuf.PushNotificationResponse))
case protobuf.ApplicationMetadataMessage_EMOJI_REACTION:
return m.unmarshalProtobufData(new(protobuf.EmojiReaction))
case protobuf.ApplicationMetadataMessage_GROUP_CHAT_INVITATION:
return m.unmarshalProtobufData(new(protobuf.GroupChatInvitation))
case protobuf.ApplicationMetadataMessage_COMMUNITY_DESCRIPTION:
return m.unmarshalProtobufData(new(protobuf.CommunityDescription))
case protobuf.ApplicationMetadataMessage_COMMUNITY_INVITATION:
return m.unmarshalProtobufData(new(protobuf.CommunityInvitation))
case protobuf.ApplicationMetadataMessage_COMMUNITY_REQUEST_TO_JOIN:
return m.unmarshalProtobufData(new(protobuf.CommunityRequestToJoin))
case protobuf.ApplicationMetadataMessage_EDIT_MESSAGE:
return m.unmarshalProtobufData(new(protobuf.EditMessage))
case protobuf.ApplicationMetadataMessage_DELETE_MESSAGE:
return m.unmarshalProtobufData(new(protobuf.DeleteMessage))
case protobuf.ApplicationMetadataMessage_STATUS_UPDATE:
return m.unmarshalProtobufData(new(protobuf.StatusUpdate))
case protobuf.ApplicationMetadataMessage_PUSH_NOTIFICATION_REGISTRATION:
// This message is a bit different as it's encrypted, so we pass it straight through
v := reflect.ValueOf(m.UnwrappedPayload)
m.ParsedMessage = &v
return nil
case protobuf.ApplicationMetadataMessage_CHAT_IDENTITY:
return m.unmarshalProtobufData(new(protobuf.ChatIdentity))
case protobuf.ApplicationMetadataMessage_ANONYMOUS_METRIC_BATCH:
return m.unmarshalProtobufData(new(protobuf.AnonymousMetricBatch))
}
return nil
}
func (m *StatusMessage) unmarshalProtobufData(pb proto.Message) error {
var ptr proto.Message
rv := reflect.ValueOf(pb)
if rv.Kind() == reflect.Ptr {
ptr = pb
} else {
ptr = rv.Addr().Interface().(proto.Message)
}
err := proto.Unmarshal(m.UnwrappedPayload, ptr)
if err != nil {
m.ParsedMessage = nil
log.Printf("[message::DecodeMessage] could not decode %T: %#x, err: %v", pb, m.Hash, err.Error())
} else {
rv = reflect.ValueOf(ptr)
elem := rv.Elem()
m.ParsedMessage = &elem
return nil
}
return nil
}