status-go/services/wallet/history/service_test.go

480 lines
9.8 KiB
Go
Raw Normal View History

feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
package history
import (
"errors"
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
"math/big"
"reflect"
"sync"
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
"testing"
"time"
"github.com/stretchr/testify/require"
"go.uber.org/mock/gomock"
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
"github.com/ethereum/go-ethereum/common"
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/ethereum/go-ethereum/event"
gethrpc "github.com/ethereum/go-ethereum/rpc"
"github.com/status-im/status-go/appdatabase"
"github.com/status-im/status-go/multiaccounts/accounts"
"github.com/status-im/status-go/rpc"
"github.com/status-im/status-go/services/accounts/accountsevent"
"github.com/status-im/status-go/t/helpers"
"github.com/status-im/status-go/t/utils"
"github.com/status-im/status-go/transactions/fake"
"github.com/status-im/status-go/walletdatabase"
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
)
func Test_entriesToDataPoints(t *testing.T) {
type args struct {
data []*entry
}
tests := []struct {
name string
args args
want []*DataPoint
wantErr bool
}{
{
name: "zeroAllChainsSameTimestamp",
args: args{
data: []*entry{
{
chainID: 1,
balance: big.NewInt(0),
timestamp: 1,
block: big.NewInt(1),
},
{
chainID: 2,
balance: big.NewInt(0),
timestamp: 1,
block: big.NewInt(5),
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(0)),
Timestamp: 1,
},
},
wantErr: false,
},
{
name: "oneZeroAllChainsDifferentTimestamp",
args: args{
data: []*entry{
{
chainID: 2,
balance: big.NewInt(0),
timestamp: 1,
block: big.NewInt(1),
},
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 2,
block: big.NewInt(2),
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(0)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(2)),
Timestamp: 2,
},
},
wantErr: false,
},
{
name: "nonZeroAllChainsDifferentTimestamp",
args: args{
data: []*entry{
{
chainID: 2,
balance: big.NewInt(1),
timestamp: 1,
},
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 2,
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(1)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(3)),
Timestamp: 2,
},
},
wantErr: false,
},
{
name: "sameChainDifferentTimestamp",
args: args{
data: []*entry{
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 1,
block: big.NewInt(1),
},
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 2,
block: big.NewInt(2),
},
{
chainID: 1,
balance: big.NewInt(0),
timestamp: 3,
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(1)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(2)),
Timestamp: 2,
},
{
Balance: (*hexutil.Big)(big.NewInt(0)),
Timestamp: 3,
},
},
wantErr: false,
},
{
name: "sameChainDifferentTimestampOtherChainsEmpty",
args: args{
data: []*entry{
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 1,
block: big.NewInt(1),
},
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 2,
block: big.NewInt(2),
},
{
chainID: 2,
balance: big.NewInt(0),
timestamp: 2,
block: big.NewInt(2),
},
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 3,
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(1)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(2)),
Timestamp: 2,
},
{
Balance: (*hexutil.Big)(big.NewInt(2)),
Timestamp: 3,
},
},
wantErr: false,
},
{
name: "onlyEdgePointsOnManyChainsWithPadding",
args: args{
data: []*entry{
// Left edge - same timestamp
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 1,
},
{
chainID: 2,
balance: big.NewInt(2),
timestamp: 1,
},
{
chainID: 3,
balance: big.NewInt(3),
timestamp: 1,
},
// Padding
{
chainID: 3,
balance: big.NewInt(3),
timestamp: 2,
},
{
chainID: 3,
balance: big.NewInt(3),
timestamp: 3,
},
{
chainID: 3,
balance: big.NewInt(3),
timestamp: 4,
},
// Right edge - same timestamp
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 5,
},
{
chainID: 2,
balance: big.NewInt(2),
timestamp: 5,
},
{
chainID: 3,
balance: big.NewInt(3),
timestamp: 5,
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(6)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(6)),
Timestamp: 2,
},
{
Balance: (*hexutil.Big)(big.NewInt(6)),
Timestamp: 3,
},
{
Balance: (*hexutil.Big)(big.NewInt(6)),
Timestamp: 4,
},
{
Balance: (*hexutil.Big)(big.NewInt(6)),
Timestamp: 5,
},
},
wantErr: false,
},
{
name: "multipleAddresses",
args: args{
data: []*entry{
{
chainID: 2,
balance: big.NewInt(5),
timestamp: 1,
address: common.Address{1},
},
{
chainID: 1,
balance: big.NewInt(6),
timestamp: 1,
address: common.Address{2},
},
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 2,
address: common.Address{1},
},
// padding - duplicate last point, just update timestamp
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 3,
address: common.Address{1},
},
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 4,
address: common.Address{1},
},
{
chainID: 1,
balance: big.NewInt(1),
timestamp: 5,
address: common.Address{1},
},
// real points
{
chainID: 1,
balance: big.NewInt(2),
timestamp: 6,
address: common.Address{2},
},
{
chainID: 1,
balance: big.NewInt(4),
timestamp: 7,
address: common.Address{2},
},
},
},
want: []*DataPoint{
{
Balance: (*hexutil.Big)(big.NewInt(11)),
Timestamp: 1,
},
{
Balance: (*hexutil.Big)(big.NewInt(12)),
Timestamp: 2,
},
// padding
{
Balance: (*hexutil.Big)(big.NewInt(12)),
Timestamp: 3,
},
{
Balance: (*hexutil.Big)(big.NewInt(12)),
Timestamp: 4,
},
{
Balance: (*hexutil.Big)(big.NewInt(12)),
Timestamp: 5,
},
// real points
{
Balance: (*hexutil.Big)(big.NewInt(8)),
Timestamp: 6,
},
{
Balance: (*hexutil.Big)(big.NewInt(10)),
Timestamp: 7,
},
},
wantErr: false,
},
}
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := entriesToDataPoints(tt.args.data)
if (err != nil) != tt.wantErr {
t.Errorf("entriesToDataPoints() name: %s, error = %v, wantErr = %v", tt.name, err, tt.wantErr)
return
}
if !reflect.DeepEqual(got, tt.want) {
t.Errorf("entriesToDataPoints() name: %s, got: %v, want: %v", tt.name, got, tt.want)
}
feat: retrieve balance history for tokens and cache it to DB Extends wallet module with the history package with the following components: BalanceDB (balance_db.go) - Keeps track of balance information (token count, block, block timestamp) for a token identity (chain, address, currency) - The cached data is stored in `balance_history` table. - Uniqueness constrained is enforced by the `balance_history_identify_entry` UNIQUE index. - Optimal DB fetching is ensured by the `balance_history_filter_entries` index Balance (balance.go) - Provides two stages: - Fetch of balance history using RPC calls (Balance.update function) - Retrieving of cached balance data from the DB it exists (Balance.get function) - Fetching and retrieving of data is done for specific time intervals defined by TimeInterval "enumeration" - Update process is done for a token identity by the Balance.Update function - The granularity of data points returned is defined by the constant increment step define in `timeIntervalToStride` for each time interval. - The `blocksStride` values have a common divisor to have cache hit between time intervals. Service (service.go) - Main APIs - StartBalanceHistory: Regularly updates balance history for all enabled networks, available accounts and provided tokens. - GetBalanceHistory: retrieves cached token count for a token identity (chain, address, currency) for multiple chains - UpdateVisibleTokens: will set the list of tokens to have historical balance fetched. This is a simplification to limit tokens to a small list that make sense Fetch balance history for ECR20 tokens - Add token.Manager.GetTokenBalanceAt to fetch balance of a specific block number of ECR20. - Add tokenChainClientSource concrete implementation of DataSource to fetch balance of ECR20 tokens. - Chose the correct DataSource implementation based on the token "is native" property. Tests Tests are implemented using a mock of `DataSource` interface used to intercept the RPC calls. Notes: - the timestamp used for retrieving block balance is constant Closes status-desktop: #8175, #8226, #8862
2022-11-15 12:14:41 +00:00
})
}
}
func Test_removeBalanceHistoryOnEventAccountRemoved(t *testing.T) {
appDB, err := helpers.SetupTestMemorySQLDB(appdatabase.DbInitializer{})
require.NoError(t, err)
walletDB, err := helpers.SetupTestMemorySQLDB(walletdatabase.DbInitializer{})
require.NoError(t, err)
accountsDB, err := accounts.NewDB(appDB)
require.NoError(t, err)
address := common.HexToAddress("0x1234")
accountFeed := event.Feed{}
walletFeed := event.Feed{}
chainID := uint64(1)
txServiceMockCtrl := gomock.NewController(t)
server, _ := fake.NewTestServer(txServiceMockCtrl)
client := gethrpc.DialInProc(server)
config := rpc.ClientConfig{
Client: client,
UpstreamChainID: chainID,
Networks: nil,
DB: appDB,
WalletFeed: nil,
ProviderConfigs: nil,
}
rpcClient, _ := rpc.NewClient(config)
rpcClient.UpstreamChainID = chainID
service := NewService(walletDB, accountsDB, &accountFeed, &walletFeed, rpcClient, nil, nil, nil)
// Insert balances for address
database := service.balance.db
err = database.add(&entry{
chainID: chainID,
address: address,
block: big.NewInt(1),
balance: big.NewInt(1),
timestamp: 1,
tokenSymbol: "ETH",
})
require.NoError(t, err)
err = database.add(&entry{
chainID: chainID,
address: address,
block: big.NewInt(2),
balance: big.NewInt(2),
tokenSymbol: "ETH",
timestamp: 2,
})
require.NoError(t, err)
entries, err := database.getNewerThan(&assetIdentity{chainID, []common.Address{address}, "ETH"}, 0)
require.NoError(t, err)
require.Len(t, entries, 2)
// Start service
service.startAccountWatcher()
// Watching accounts must start before sending event.
// To avoid running goroutine immediately and let the controller subscribe first,
// use any delay.
group := sync.WaitGroup{}
group.Add(1)
go func() {
defer group.Done()
time.Sleep(1 * time.Millisecond)
accountFeed.Send(accountsevent.Event{
Type: accountsevent.EventTypeRemoved,
Accounts: []common.Address{address},
})
err := utils.Eventually(func() error {
entries, err := database.getNewerThan(&assetIdentity{1, []common.Address{address}, "ETH"}, 0)
if err == nil && len(entries) == 0 {
return nil
}
return errors.New("data is not removed")
}, 100*time.Millisecond, 10*time.Millisecond)
require.NoError(t, err)
}()
group.Wait()
// Stop service
txServiceMockCtrl.Finish()
server.Stop()
service.stopAccountWatcher()
}