🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
package appmetrics
|
|
|
|
|
|
|
|
import (
|
|
|
|
"database/sql"
|
|
|
|
"encoding/json"
|
|
|
|
"errors"
|
|
|
|
|
|
|
|
"strings"
|
|
|
|
|
|
|
|
"github.com/xeipuuv/gojsonschema"
|
|
|
|
)
|
|
|
|
|
|
|
|
type AppMetricEventType string
|
|
|
|
|
|
|
|
// Value is `json.RawMessage` so we can send any json shape, including strings
|
|
|
|
// Validation is handled using JSON schemas defined in validators.go, instead of Golang structs
|
|
|
|
type AppMetric struct {
|
|
|
|
Event AppMetricEventType `json:"event"`
|
|
|
|
Value json.RawMessage `json:"value"`
|
|
|
|
AppVersion string `json:"app_version"`
|
|
|
|
OS string `json:"os"`
|
2021-04-12 13:46:11 +00:00
|
|
|
SessionID string `json:"session_id"`
|
|
|
|
CreatedAt string `json:"created_at"`
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
type AppMetricValidationError struct {
|
|
|
|
Metric AppMetric
|
|
|
|
Errors []gojsonschema.ResultError
|
|
|
|
}
|
|
|
|
|
|
|
|
const (
|
|
|
|
// status-react navigation events
|
2021-04-12 12:25:53 +00:00
|
|
|
NavigateTo AppMetricEventType = "navigate-to"
|
|
|
|
ScreensOnWillFocus AppMetricEventType = "screens/on-will-focus"
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
)
|
|
|
|
|
|
|
|
// EventSchemaMap Every event should have a schema attached
|
|
|
|
var EventSchemaMap = map[AppMetricEventType]interface{}{
|
2021-04-12 12:25:53 +00:00
|
|
|
NavigateTo: NavigateToCofxSchema,
|
|
|
|
ScreensOnWillFocus: NavigateToCofxSchema,
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewDB(db *sql.DB) *Database {
|
|
|
|
return &Database{db: db}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Database sql wrapper for operations with browser objects.
|
|
|
|
type Database struct {
|
|
|
|
db *sql.DB
|
|
|
|
}
|
|
|
|
|
|
|
|
// Close closes database.
|
|
|
|
func (db Database) Close() error {
|
|
|
|
return db.db.Close()
|
|
|
|
}
|
|
|
|
|
|
|
|
func jsonschemaErrorsToError(validationErrors []AppMetricValidationError) error {
|
|
|
|
var fieldErrors []string
|
|
|
|
|
|
|
|
for _, appMetricValidationError := range validationErrors {
|
|
|
|
metric := appMetricValidationError.Metric
|
|
|
|
errors := appMetricValidationError.Errors
|
|
|
|
|
|
|
|
var errorDesc string = "Error in event: " + string(metric.Event) + " - "
|
|
|
|
for _, e := range errors {
|
|
|
|
errorDesc = errorDesc + "value." + e.Context().String() + ":" + e.Description()
|
|
|
|
}
|
|
|
|
fieldErrors = append(fieldErrors, errorDesc)
|
|
|
|
}
|
|
|
|
|
|
|
|
return errors.New(strings.Join(fieldErrors[:], "/ "))
|
|
|
|
}
|
|
|
|
|
|
|
|
func (db *Database) ValidateAppMetrics(appMetrics []AppMetric) (err error) {
|
|
|
|
var calculatedErrors []AppMetricValidationError
|
|
|
|
for _, metric := range appMetrics {
|
|
|
|
schema := EventSchemaMap[metric.Event]
|
|
|
|
|
|
|
|
if schema == nil {
|
|
|
|
return errors.New("No schema defined for: " + string(metric.Event))
|
|
|
|
}
|
|
|
|
|
|
|
|
schemaLoader := gojsonschema.NewGoLoader(schema)
|
|
|
|
valLoader := gojsonschema.NewStringLoader(string(metric.Value))
|
|
|
|
res, err := gojsonschema.Validate(schemaLoader, valLoader)
|
|
|
|
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
// validate all metrics and save errors
|
|
|
|
if !res.Valid() {
|
|
|
|
calculatedErrors = append(calculatedErrors, AppMetricValidationError{metric, res.Errors()})
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if len(calculatedErrors) > 0 {
|
|
|
|
return jsonschemaErrorsToError(calculatedErrors)
|
|
|
|
}
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
2021-04-12 13:46:11 +00:00
|
|
|
func (db *Database) SaveAppMetrics(appMetrics []AppMetric, sessionID string) (err error) {
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
var (
|
|
|
|
tx *sql.Tx
|
|
|
|
insert *sql.Stmt
|
|
|
|
)
|
|
|
|
|
|
|
|
// make sure that the shape of the metric is same as expected
|
|
|
|
err = db.ValidateAppMetrics(appMetrics)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
// start txn
|
|
|
|
tx, err = db.db.Begin()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
defer func() {
|
|
|
|
if err == nil {
|
|
|
|
err = tx.Commit()
|
|
|
|
return
|
|
|
|
}
|
|
|
|
_ = tx.Rollback()
|
|
|
|
}()
|
|
|
|
|
2021-04-12 13:46:11 +00:00
|
|
|
insert, err = tx.Prepare("INSERT INTO app_metrics (event, value, app_version, operating_system, session_id) VALUES (?, ?, ?, ?, ?)")
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, metric := range appMetrics {
|
2021-04-12 13:46:11 +00:00
|
|
|
_, err = insert.Exec(metric.Event, metric.Value, metric.AppVersion, metric.OS, sessionID)
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
if err != nil {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
func (db *Database) GetAppMetrics(limit int, offset int) (appMetrics []AppMetric, err error) {
|
2021-04-12 13:46:11 +00:00
|
|
|
rows, err := db.db.Query("SELECT event, value, app_version, operating_system, session_id, created_at FROM app_metrics LIMIT ? OFFSET ?", limit, offset)
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
defer rows.Close()
|
|
|
|
for rows.Next() {
|
|
|
|
metric := AppMetric{}
|
2021-04-12 13:46:11 +00:00
|
|
|
err := rows.Scan(
|
|
|
|
&metric.Event, &metric.Value,
|
|
|
|
&metric.AppVersion, &metric.OS,
|
|
|
|
&metric.SessionID, &metric.CreatedAt,
|
|
|
|
)
|
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 12:39:28 +00:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
appMetrics = append(appMetrics, metric)
|
|
|
|
}
|
|
|
|
return appMetrics, nil
|
|
|
|
}
|