2020-03-19 10:28:21 +00:00
|
|
|
import Service, { inject as service } from '@ember/service';
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
import { runInDebug } from '@ember/debug';
|
2020-07-17 13:42:45 +00:00
|
|
|
import { proxy } from 'consul-ui/utils/dom/event-source';
|
2020-11-30 15:05:16 +00:00
|
|
|
import { schedule } from '@ember/runloop';
|
2020-03-19 10:28:21 +00:00
|
|
|
|
|
|
|
import MultiMap from 'mnemonist/multi-map';
|
|
|
|
|
|
|
|
// TODO: Expose sizes of things via env vars
|
|
|
|
|
|
|
|
// caches cursors and previous events when the EventSources are destroyed
|
2020-05-11 15:37:11 +00:00
|
|
|
let cache = null;
|
2020-03-19 10:28:21 +00:00
|
|
|
// keeps a record of currently in use EventSources
|
2020-05-11 15:37:11 +00:00
|
|
|
let sources = null;
|
2020-03-19 10:28:21 +00:00
|
|
|
// keeps a count of currently in use EventSources
|
2020-05-11 15:37:11 +00:00
|
|
|
let usage = null;
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
class URI {
|
|
|
|
constructor(uri) {
|
|
|
|
this.uri = uri;
|
|
|
|
}
|
|
|
|
toString() {
|
|
|
|
return this.uri;
|
|
|
|
}
|
|
|
|
}
|
2020-11-09 09:25:35 +00:00
|
|
|
export default class DataSourceService extends Service {
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
@service('dom') dom;
|
|
|
|
@service('encoder') encoder;
|
|
|
|
@service('data-source/protocols/http') consul;
|
|
|
|
@service('data-source/protocols/local-storage') settings;
|
2020-11-09 09:25:35 +00:00
|
|
|
|
|
|
|
init() {
|
|
|
|
super.init(...arguments);
|
2020-05-27 10:23:21 +00:00
|
|
|
cache = new Map();
|
|
|
|
sources = new Map();
|
|
|
|
usage = new MultiMap(Set);
|
2020-05-11 15:37:11 +00:00
|
|
|
this._listeners = this.dom.listeners();
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
resetCache() {
|
2020-03-19 10:28:21 +00:00
|
|
|
cache = new Map();
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
willDestroy() {
|
2020-11-30 15:05:16 +00:00
|
|
|
// the will-destroy helper will fire AFTER services have had willDestroy
|
|
|
|
// called on them, schedule any destroying to fire after the final render
|
|
|
|
schedule('afterRender', () => {
|
|
|
|
this._listeners.remove();
|
|
|
|
sources.forEach(function(item) {
|
|
|
|
item.close();
|
|
|
|
});
|
|
|
|
cache = null;
|
|
|
|
sources = null;
|
|
|
|
usage.clear();
|
|
|
|
usage = null;
|
2020-04-21 15:49:11 +00:00
|
|
|
});
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
source(cb, attrs) {
|
2020-07-17 13:42:45 +00:00
|
|
|
const src = cb(this.encoder.uriTag());
|
|
|
|
return new Promise((resolve, reject) => {
|
|
|
|
const ref = {};
|
|
|
|
const source = this.open(src, ref, true);
|
|
|
|
source.configuration.ref = ref;
|
|
|
|
const remove = this._listeners.add(source, {
|
|
|
|
message: e => {
|
|
|
|
remove();
|
|
|
|
// the source only gets wrapped in the proxy
|
|
|
|
// after the first message
|
|
|
|
// but the proxy itself is resolve to the route
|
|
|
|
resolve(proxy(e.target, e.data));
|
|
|
|
},
|
|
|
|
error: e => {
|
|
|
|
remove();
|
|
|
|
this.close(source, ref);
|
|
|
|
reject(e.error);
|
|
|
|
},
|
|
|
|
});
|
|
|
|
if (typeof source.getCurrentEvent() !== 'undefined') {
|
|
|
|
source.dispatchEvent(source.getCurrentEvent());
|
|
|
|
}
|
|
|
|
});
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
unwrap(src, ref) {
|
2020-07-17 13:42:45 +00:00
|
|
|
const source = src._source;
|
|
|
|
usage.set(source, ref);
|
|
|
|
usage.remove(source, source.configuration.ref);
|
|
|
|
delete source.configuration.ref;
|
|
|
|
return source;
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
uri(str) {
|
|
|
|
return new URI(str);
|
|
|
|
}
|
|
|
|
|
2020-11-09 09:25:35 +00:00
|
|
|
open(uri, ref, open = false) {
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
if (!(uri instanceof URI) && typeof uri !== 'string') {
|
2020-07-17 13:42:45 +00:00
|
|
|
return this.unwrap(uri, ref);
|
|
|
|
}
|
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117)
Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper.
Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not.
Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above.
The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors.
Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated.
Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call.
The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞
Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
|
|
|
runInDebug(
|
|
|
|
_ => {
|
|
|
|
if(!(uri instanceof URI)) {
|
|
|
|
console.error(new Error(`DataSource '${uri}' does not use the uri helper. Please ensure you use the uri helper to ensure correct encoding`))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
);
|
|
|
|
uri = uri.toString();
|
2020-03-19 10:28:21 +00:00
|
|
|
let source;
|
|
|
|
// Check the cache for an EventSource that is already being used
|
|
|
|
// for this uri. If we don't have one, set one up.
|
|
|
|
if (uri.indexOf('://') === -1) {
|
|
|
|
uri = `consul://${uri}`;
|
|
|
|
}
|
2020-07-17 13:42:45 +00:00
|
|
|
let [providerName, pathname] = uri.split('://');
|
|
|
|
const provider = this[providerName];
|
2020-03-19 10:28:21 +00:00
|
|
|
if (!sources.has(uri)) {
|
|
|
|
let configuration = {};
|
|
|
|
if (cache.has(uri)) {
|
|
|
|
configuration = cache.get(uri);
|
|
|
|
}
|
2020-07-17 13:42:45 +00:00
|
|
|
configuration.uri = uri;
|
2020-03-19 10:28:21 +00:00
|
|
|
source = provider.source(pathname, configuration);
|
2020-07-17 13:42:45 +00:00
|
|
|
const remove = this._listeners.add(source, {
|
2020-03-19 10:28:21 +00:00
|
|
|
close: e => {
|
2020-07-17 13:42:45 +00:00
|
|
|
// a close could be fired either by:
|
|
|
|
// 1. A non-blocking query leaving the page
|
|
|
|
// 2. A non-blocking query responding
|
|
|
|
// 3. A blocking query responding when is in a closing state
|
|
|
|
// 3. A non-blocking query or a blocking query being cancelled
|
2020-03-19 10:28:21 +00:00
|
|
|
const source = e.target;
|
2020-05-27 10:23:21 +00:00
|
|
|
const event = source.getCurrentEvent();
|
|
|
|
const cursor = source.configuration.cursor;
|
|
|
|
// only cache data if we have any
|
|
|
|
if (typeof event !== 'undefined' && typeof cursor !== 'undefined') {
|
|
|
|
cache.set(uri, {
|
2020-07-09 09:08:47 +00:00
|
|
|
currentEvent: event,
|
|
|
|
cursor: cursor,
|
2020-05-27 10:23:21 +00:00
|
|
|
});
|
|
|
|
}
|
2020-03-19 10:28:21 +00:00
|
|
|
// the data is cached delete the EventSource
|
2020-07-09 09:08:47 +00:00
|
|
|
if (!usage.has(source)) {
|
2020-07-17 13:42:45 +00:00
|
|
|
// A non-blocking query could close but still be on the page
|
2020-07-09 09:08:47 +00:00
|
|
|
sources.delete(uri);
|
|
|
|
}
|
2020-07-17 13:42:45 +00:00
|
|
|
remove();
|
2020-03-19 10:28:21 +00:00
|
|
|
},
|
|
|
|
});
|
|
|
|
sources.set(uri, source);
|
|
|
|
} else {
|
|
|
|
source = sources.get(uri);
|
2020-07-17 13:42:45 +00:00
|
|
|
// bump to the end of the list
|
|
|
|
sources.delete(uri);
|
|
|
|
sources.set(uri, source);
|
2020-03-19 10:28:21 +00:00
|
|
|
}
|
2020-07-09 09:08:47 +00:00
|
|
|
// only open if its not already being used
|
|
|
|
// in the case of blocking queries being disabled
|
|
|
|
// you may want to specifically force an open
|
|
|
|
// if blocking queries are enabled then opening an already
|
|
|
|
// open blocking query does nothing
|
2020-07-17 13:42:45 +00:00
|
|
|
if (!usage.has(source) || source.readyState > 1 || open) {
|
2020-07-09 09:08:47 +00:00
|
|
|
source.open();
|
|
|
|
}
|
2020-03-19 10:28:21 +00:00
|
|
|
// set/increase the usage counter
|
|
|
|
usage.set(source, ref);
|
|
|
|
return source;
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
close(source, ref) {
|
2020-07-17 13:42:45 +00:00
|
|
|
// this close is called when the source has either left the page
|
|
|
|
// or in the case of a proxied source, it errors
|
2020-03-19 10:28:21 +00:00
|
|
|
if (source) {
|
|
|
|
// decrease the usage counter
|
|
|
|
usage.remove(source, ref);
|
|
|
|
// if the EventSource is no longer being used
|
|
|
|
// close it (data caching is dealt with by the above 'close' event listener)
|
|
|
|
if (!usage.has(source)) {
|
|
|
|
source.close();
|
2020-07-17 13:42:45 +00:00
|
|
|
if (source.readyState === 2) {
|
|
|
|
// in the case that a non-blocking query is on the page
|
|
|
|
// and it has already responded and has therefore been cached
|
|
|
|
// but not removed itself from sources
|
|
|
|
// delete from sources
|
|
|
|
sources.delete(source.configuration.uri);
|
|
|
|
}
|
2020-03-19 10:28:21 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
closed() {
|
2020-07-17 13:42:45 +00:00
|
|
|
// anything that is closed or closing
|
|
|
|
return [...sources.entries()]
|
|
|
|
.filter(([key, item]) => {
|
|
|
|
return item.readyState > 1;
|
|
|
|
})
|
|
|
|
.map(item => item[0]);
|
2020-11-09 09:25:35 +00:00
|
|
|
}
|
|
|
|
}
|