🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
package appmetrics
|
|
|
|
|
|
|
|
import (
|
|
|
|
"encoding/json"
|
2021-09-09 14:28:54 +00:00
|
|
|
"fmt"
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
"testing"
|
2021-09-01 12:02:18 +00:00
|
|
|
"time"
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
|
|
|
|
"github.com/status-im/status-go/appdatabase"
|
2023-08-11 11:28:45 +00:00
|
|
|
"github.com/status-im/status-go/t/helpers"
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
|
|
|
|
"github.com/stretchr/testify/require"
|
|
|
|
)
|
|
|
|
|
|
|
|
func setupTestDB(t *testing.T) (*Database, func()) {
|
2023-08-11 11:28:45 +00:00
|
|
|
db, cleanup, err := helpers.SetupTestSQLDB(appdatabase.DbInitializer{}, "appmetrics-tests")
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
require.NoError(t, err)
|
2023-08-11 11:28:45 +00:00
|
|
|
return NewDB(db), func() { require.NoError(t, cleanup()) }
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
func TestSaveAppMetrics(t *testing.T) {
|
2021-04-12 13:46:11 +00:00
|
|
|
sessionID := "rand-omse-ssid"
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
db, stop := setupTestDB(t)
|
|
|
|
defer stop()
|
|
|
|
|
|
|
|
// we need backticks (``) for value because it is expected by gojsonschema
|
|
|
|
// it considers text inside tics to be stringified json
|
|
|
|
appMetrics := []AppMetric{
|
2021-04-12 12:25:53 +00:00
|
|
|
{Event: NavigateTo, Value: json.RawMessage(`{"view_id": "some-view-id", "params": {"screen": "login"}}`), OS: "android", AppVersion: "1.11"},
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
}
|
|
|
|
|
2021-04-12 13:46:11 +00:00
|
|
|
err := db.SaveAppMetrics(appMetrics, sessionID)
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
|
2021-05-21 08:34:28 +00:00
|
|
|
appMetricsPage, err := db.GetAppMetrics(10, 0)
|
|
|
|
res := appMetricsPage.AppMetrics
|
|
|
|
count := appMetricsPage.TotalCount
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
require.NoError(t, err)
|
2021-04-12 13:46:11 +00:00
|
|
|
require.Equal(t, appMetrics[0].Event, res[0].Event)
|
|
|
|
require.Equal(t, appMetrics[0].Value, res[0].Value)
|
|
|
|
require.Equal(t, appMetrics[0].OS, res[0].OS)
|
|
|
|
require.Equal(t, appMetrics[0].AppVersion, res[0].AppVersion)
|
2021-09-01 12:02:18 +00:00
|
|
|
require.False(t, res[0].Processed)
|
2021-04-12 13:46:11 +00:00
|
|
|
require.NotNil(t, res[0].CreatedAt)
|
2021-05-21 08:34:28 +00:00
|
|
|
require.Equal(t, count, 1)
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
}
|
2021-09-01 12:02:18 +00:00
|
|
|
|
|
|
|
func TestDatabase_GetUnprocessedMetrics(t *testing.T) {
|
|
|
|
db, stop := setupTestDB(t)
|
|
|
|
defer stop()
|
|
|
|
|
|
|
|
var uam []AppMetric
|
|
|
|
metricsPerSession := 10
|
|
|
|
unprocessedMetricsPerSession := 5
|
|
|
|
numberOfSessions := 3
|
|
|
|
numberOfSessionSaves := 5
|
|
|
|
|
|
|
|
for i := 0; i < numberOfSessionSaves; i++ {
|
|
|
|
for ii := 1; ii < numberOfSessions+1; ii++ {
|
2021-09-09 14:28:54 +00:00
|
|
|
err := db.SaveAppMetrics(GenerateMetrics(metricsPerSession), "rand-omse-ssid-"+fmt.Sprint(ii))
|
2021-09-01 12:02:18 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
uam, err = db.GetUnprocessed()
|
|
|
|
require.NoError(t, err)
|
|
|
|
require.Len(t, uam, unprocessedMetricsPerSession*ii+(i*numberOfSessions*unprocessedMetricsPerSession))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Test metrics are grouped by session_id
|
|
|
|
lastSessionID := ""
|
|
|
|
sessionCount := 0
|
|
|
|
for _, m := range uam {
|
|
|
|
if lastSessionID != m.SessionID {
|
|
|
|
lastSessionID = m.SessionID
|
|
|
|
sessionCount++
|
|
|
|
}
|
|
|
|
}
|
|
|
|
require.Equal(t, numberOfSessions, sessionCount)
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestDatabase_SetProcessedMetrics(t *testing.T) {
|
|
|
|
db, stop := setupTestDB(t)
|
|
|
|
defer stop()
|
|
|
|
|
|
|
|
// Add sample data to the DB
|
|
|
|
err := db.SaveAppMetrics(GenerateMetrics(20), "rand-omse-ssid")
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Get only the unprocessed metrics
|
|
|
|
uam, err := db.GetUnprocessed()
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Extract the ids from the metrics IDs
|
|
|
|
ids := GetAppMetricsIDs(uam)
|
|
|
|
|
|
|
|
// Add some more metrics to the DB
|
|
|
|
err = db.SaveAppMetrics(GenerateMetrics(20), "rand-omse-ssid-2")
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Set metrics as processed with the given ids
|
|
|
|
err = db.SetToProcessedByIDs(ids)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Check we have the expected number of unprocessed metrics in the db
|
|
|
|
uam, err = db.GetUnprocessed()
|
|
|
|
require.NoError(t, err)
|
|
|
|
require.Len(t, uam, 10)
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestDatabase_GetUnprocessedGroupedBySession(t *testing.T) {
|
|
|
|
db, stop := setupTestDB(t)
|
|
|
|
defer stop()
|
|
|
|
|
|
|
|
// Add sample data to the DB
|
|
|
|
err := db.SaveAppMetrics(GenerateMetrics(20), "rand-omse-ssid")
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Add some more metrics to the DB
|
|
|
|
err = db.SaveAppMetrics(GenerateMetrics(20), "rand-omse-ssid-2")
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Check we have the expected number of unprocessed metrics in the db
|
|
|
|
uam, err := db.GetUnprocessedGroupedBySession()
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Check we have 2 groups / sessions
|
|
|
|
require.Len(t, uam, 2)
|
|
|
|
require.Len(t, uam["rand-omse-ssid"], 10)
|
|
|
|
require.Len(t, uam["rand-omse-ssid-2"], 10)
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestDatabase_DeleteOlderThan(t *testing.T) {
|
|
|
|
db, stop := setupTestDB(t)
|
|
|
|
defer stop()
|
|
|
|
|
|
|
|
threeHoursAgo := time.Now().Add(time.Hour * -3) // go is annoying sometimes
|
|
|
|
oneHourHence := time.Now().Add(time.Hour)
|
|
|
|
|
|
|
|
// Add sample data to the DB
|
|
|
|
err := db.SaveAppMetrics(GenerateMetrics(20), "rand-omse-ssid")
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Delete all messages older than 3 hours old
|
|
|
|
err = db.DeleteOlderThan(&threeHoursAgo)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Get all metrics from DB, none should be deleted
|
|
|
|
ams, err := db.GetAppMetrics(100, 0)
|
|
|
|
require.NoError(t, err)
|
|
|
|
require.Len(t, ams.AppMetrics, 20)
|
|
|
|
|
|
|
|
// Delete all messages older than 1 hours in the future
|
|
|
|
err = db.DeleteOlderThan(&oneHourHence)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Get all metrics from DB, all should be deleted
|
|
|
|
ams, err = db.GetAppMetrics(100, 0)
|
|
|
|
require.NoError(t, err)
|
|
|
|
require.Len(t, ams.AppMetrics, 0)
|
|
|
|
}
|