Summary:
Buck (our build system) currently starts multiple packager instances for each target and may build multiple targets in parallel. This means we're paying startup costs and are duplicating the work. This enables us to start one instance of the packager and connect to it via socket to do all the work that needs to be done.
The way this is structured:
1. SocketServer: A server that listens on a socket path that is generated based on the server options
2. SocketClient: Interfaces with the server and exposes the operations that we support as methods
3. SocketInterface: Integration point and responsible for forking off the server
Summary:
The transform step in currently the longest one in the bundling process. This adds a progress bar to track the transform progress.
{F23096660}
Summary:
There are two fs steps and it wasn't clear why. This now puts the right label:
```
[9:38:25 PM] <START> Building in-memory fs for JavaScript
[9:38:27 PM] <END> Building in-memory fs for JavaScript (2030ms)
[9:38:27 PM] <START> Building in-memory fs for Assets
[9:38:27 PM] <END> Building in-memory fs for Assets (615ms)
```
Summary:
The `BundlesLayout` will be used as a persistent index. As such, it would be easier to avoid having dependencies to `Module`, `Package`, `Asset`, etc. We're not using that information for now and if we happen to need to use it we could always fetch it using the `ModuleCache`.
Summary:
We've decided to move the syntax for asynchronously requiring async dependencies. The new syntax works better with promises and therefore withe async/await as well. The new syntax looks like this: `System.import('moduleA').then(moduleA => {...});` or if you're using async/await you could simply do:
let moduleA = await System.import('moduleA');
new moduleA().someFunction();
If you need to require multiple dependencies just do:
Promise
.all([System.import('moduleA'), System.import('moduleB')])
.then((moduleA, moduleB) => {...})
or the equivalent using async/await
Summary:
Fix error in the template string (no plus, thinks it's a function).
And bump the timeout to 30 seconds because a file is taking more than 10 seconds `js/RKJSModules/Libraries/FBComponents/FBFed/FeedStoryFragments.js`
Summary:
Since JS doesn't have the guarantee that once a bundle is loaded it will stay in memory (and this is something we actually don't want to enforce to keep memmory usage low), we need to keep track of parent/child relationships on the packager to pass it down to native. As part of this diff, we also introduced an ID for each bundle. The ID for a child bundle is shynthetized as the bundleID of the parent module + an index which gets incremented every time a new bundle is created. For instance given this tree:
a,b
c f
d e g
the ID for `d` will be `bundle.0.1.2`, the one for e will be `bundle.0.1.3` and the one for `g` will be `bundle.0.5.6`. This information will be useful to figure out which bundles need to be loaded when a `require.ensure` is re-written.
Summary:
There's been a case where Babel can hang indefinitely on a file parse/transform. Possibly related to https://github.com/babel/babel/issues/2211
This adds a timeout to transform jobs and throws an error informing the user of the offending file. The timeout interval defaults to 10 seconds, but can be changed via an option.
Summary:
D2319999 introduced a regression where we stopped waiting for the "build haste map" step to finish before we accept any requests. This makes sure that we block on that.
Need to unbreak with this, but will follow up with a test to catch this in the future.
Summary:
Currently the platform selection is controlled by the blacklist. However, since we want to use the same server instance for cross-platform development, we need this to be controlled per request.
One outstanding issue, is that the DependencyGraph class wasn't designed that way and it doesn't have a per-request state. This means that with the current design race conditions is possible. If we got a request for a different platfrom while processing the previous request, we may change the outcome of the previous request.
To fix this a larger refactor is needed. I'll follow up a diff to do that.
Finally, so I don't break the universe like last time, I'll leave it up to the RN guys to update the call sites.
Summary:
Not that at the moment a module can be present in multiple bundles, so the new API will return only one of them. In the near future we'll impose the invariant that a module can only be present in a single bundle so this API will return the exact bundle in which it is.
Summary:
Instead of using plain objects and having to convert to and from them we just use the `Module` class across the codebase.
This seems cleaner and can enforce the type as opposed to fuzzy objects.
Summary:
Introduce a Bundler capable of generating the layout of modules for a given entry point. The current algorithm is the most trivial we could come up with: (1)it puts all the sync dependencies into the same bundle and (2) each group of async dependencies with all their dependencies into a separate bundle. For async dependencies we do this recursivelly, meaning that async dependencies could have async dependencies which will end up on separate bundles as well.
The output of of the layout is an array of bundles. Each bundle is just an array for now with the dependencies in the order the requires where processed. Using this information we should be able to generate the actual bundles by using the `/path/to/entry/point.bundle` endpoint. We might change the structure of this json in the future, for instance to account for parent/child bundles relationships.
The next step will be to improve this algorithm to avoid repeating quite a bit dependencies across bundles.
Summary:
This is the first step to add support for splitting the JS bundle into multiple ones. This diff adds support for keeping track of the async dependencies each module has. To do so we introduce the following syntax:
require.ensure(['dep1', 'dep2, ..., 'depN'], callback);
Where the callback function is asynchronously invoked once all the indicated modules are loaded.
Internally, the packager keeps track of every set of async dependencies a module has. So for instance if a module looks like this:
require.ensure(['dep1'], () => {...});
require.ensure(['dep2'], () => {...});
the `Module` object will keep track of each set of dependencies separately (because we might want to put them on separate bundles).
Summary:
The word Package is overloaded, it may mean npm package, or may mean a collection of bundles. Neither is what we mean. We mean `bundle`.
This renames it and modernize some of the Bundler code.
Summary:
The cache is only used for JSTransformer at the moment. We're doing IO and some computation to get each module's name, whether is a haste or node module and it's dependencies. This work happens on startup so by caching this value we shouldbe able to reduce the start up time. Lets promote the Cache to the Packager level to be able to use it by any of the components of the packager. For now, on this diff we'll start using it to cache the mentioned fields.
Also we had to introduce the concept of fields in the cache as manually merging the date we had for each path is not possible as we're using promisses all around. With the new API, each field is a promise.
@amasad and I did some manual testing to measure the impact of this change and looks like it's saves 1 second when building the haste map (which represents 50% of the time). Overall this reduces 1 second of start up time which was currently about 8s on my mac book pro.
Summary:
- Enables async/await in .babelrc and transformer.js
- Adds regenerator to package.json. Users still need to explicitly require the regenerator runtime -- this is so that you only pay for what you use.
- Update AsyncStorage examples in UIExplorer to use async/await
- Update promise tests in UIExplorer to use async/await in addition to the promise API
Closes https://github.com/facebook/react-native/pull/1765
Github Author: James Ide <ide@jameside.com>
Summary:
Teach the resolver about platform-based resolution. The platform extension is inferred from the entry point.
It works for haste modules, as well as node-based resolution.
Summary:
[This is a preview diff for getting RN's tests to pass with a future version of jest that supports io.js and other future versions of Node. This can be merged once the diff to update jest is merged upstream and published.]
Updates the tests in small ways so they run on io.js with two updates:
- The Cache test which relies on Promises uses `runAllImmediates` for modern versions of Node because bluebird uses `setImmediate` instead of `process.nextTick` for Node >0.10.
Closes https://github.com/facebook/react-native/pull/1382
Github Author: James Ide <ide@jameside.com>
Test Plan: Run `npm test` with the latest version of jest.
Summary:
@public
Now that watchman perf issue was fixed we can enable watchman-based fs crawling which is faster than node.
This showed an existing issue with some files missing from the blacklist which I addressed.
Test Plan:
./fbrnios.sh run
click around and scroll all the apps
Summary:
@public
We have a function that detects whether a give file is to be treated as a node_modules. If so it doesn't have access to the haste module map. There is an exception to this rule which is a few modules that are allowed to do that. Currently thats react-native, react-tools, and parse.
The current implementation had a bug where if you had `react-native` (or react-tools etc) in the name before the actual package root then the detection will be off. This fixes the problem by starting from the `lastIndexOf('node_modules')` directory, that way nothing confuses us.
Test Plan:
./runJestTests.sh
export OSS, patch, run e2e test
Summary:
@public
The current output of console.log is extremely bad. If you pass NaN, it shows up as null (super confusing I know -_-), if you pass a cyclical object, it just says that it is cyclic and that's it. It doesn't print up the first few levels which are NOT cyclical and would be really helpful.
It turns out that nodejs console.log pretty printer is really awesome and can be easily extracted as a few hundred lines. This is going to be such a productivity boost that I think it's the right tradeoff to embed it like this
Test Plan:
```
var a = {kikoo: {lol: 1}}
a.kikoo.nice = a;
console.log(a);
> { kikoo: { lol: 1, nice: [Circular] } }
console.log(NaN)
> NaN
```
Summary:
@public
corrected small typo in the 500 response from the packager server
Test Plan: add throw to promise function prior to error handler, run packager, cache a bundle with bundle extension URI, open /debug/packages, see clean 500 error
Summary:
@public
Fixes#773, #1055
The resolver was getting a bit unwieldy because a lot has changed since the initial writing (porting node-haste).
This also splits up a large complex file into the following:
* Makes use of classes: Module, AssetModule, Package, and AssetModule_DEPRECATED (`image!` modules)
* DependencyGraph is lazy for everything that isn't haste modules and packages (need to read ahead of time)
* Lazy makes it fast, easier to reason about, and easier to add new loaders
* Has a centralized filesystem wrapper: fast-fs (ffs)
* ffs is async and lazy for any read operation and sync for directory/file lookup which makes it fast
* we can easily drop in different adapters for ffs to be able to build up the tree: watchman, git ls-files, etc
* use es6 for classes and easier to read promise-based code
Follow up diffs will include:
* Using new types (Module, AssetModule etc) in the rest of the codebase (currently we convert to plain object which is a bit of a hack)
* using watchman to build up the fs
* some caching at the object creation level (we are recreating Modules and Packages many times, we can cache them)
* A plugin system for loaders (e.g. @tadeuzagallo wants to add a native module loader)
Test Plan:
* ./runJestTests.sh react-packager
* ./runJestTests.sh PackagerIntegration
* Export open source and run the e2e test
* reset cache
* ./fbrnios.sh run and click around
Summary:
@public
We cached based on url, which wasn't unique becuase some options would be defaulted. This was obvious when starting the server via fbrnios which tries to warmup the bundle.
And then when the device woke up it will send a request (that is identical in reality) but would miss the cache.
This changes the cache key into a JSON stringification of the options.
Test Plan:
* ./runJestTests.sh
* ./fbrnios.sh run
Summary:
@public
Fixes#1431Fixes#1005
Files with no newlines and a comment at the end of the file would've caused a syntax error in the bundle:
```js
__d('module', function() {
hi();
// wow })
```
This fixes the issue by inserting a new lines before `})`.
Test Plan:
* ./runJestTests.sh
* ./runJestTests.sh PackagerIntegration
* open app to the playground app
* add an error
* observe that the redbox has the correct lines
Summary:
@public
Fixes#773
This fixes `.json` name resolution. And also reads `package.json` when doing a directory module resolution.
The algorithm can be found here: https://nodejs.org/api/modules.html
I'll probably start including the node (or browserify) modules test in later diffs to make sure we're fully compliant.
Test Plan:
* ./runJestTests.sh
* ./runJestTests.sh PackagerIntegration
* open playground and require a json file
* test redbox
Summary:
@public
Fixes issue #1055
For some historical reason we used to strip the extension of the module name before passing it to `resolveDependency` which is completly capable of handling all kinds of names. The fix is one line, but added a few tests for this.
Test Plan:
* ./runJestTests.sh
* ./runJestTests.sh PacakgerIntegration
* Open app and click around
Summary:
@public
The packager's resolver started out imitating node-haste, which meant that we didn't support nested modules. Now this is a problem. Bigger projects are bound to have versions of different versions of the same package at different levels of the dependency tree. This
makes loading dependencies lazy for node_modules and implements the node resolution algorithm. However, it also mantains that some
modules are still "haste" format, which currently defaults to "react-native" and "react-tools".
Finally, this means ~5 seconds speed up on every server start. This should also have a big impact on open source users with projects with big node_modules.
Test Plan:
1- test the app with --reset-cache
2- click around test and production apps
3- update the OSS library
4- create a new project
5- npm install multiple modules
6- create some version conflict in your project
7- make sure we do the "right" thing
8- test file changes to make sure it works
Summary:
@public
Previously, we had to use errors as a property on the result object because there was no way to pass custom props between
the child worker and the parent. This has been fixed in node-worker-farm (D2092153) and now we can use regular errors.
This also adapts the transformer to babel-specific errors. Generic errors, however, should still work and render readable
info.
Additionally, I deprecated, but maintained backwards compatiblity for people in OSS that are using custom transformers.
Test Plan:
1. `./runJestTests.sh`
2. `./runJestTests.sh PackagerIntegration`
3. open the playground app
4. Add a syntax error. Say `1=x` somewhere in the file
5. Reload and see error message 'SyntaxError <filename> description (line:col)'
6. Make sure that the stack trace is clickable and it attempts to open the editor to the location
Summary:
@public
1. Default to first class support of popular image formats
2. Add tests to make sure we support other than png
Test Plan:
1. ./runJestTests.sh
2. Add test.png and test.jpg images in the Playground app dir
3. require both images and render then in the playground app
4. they render
Summary:
@public
Shouldn't confuse the cache from files transformed by different transformers. This takes into account the transformer in the cache hash name.
Test Plan:
* start server with --babel
* generate bundle
* start server with --jstransform
* generate bundle
* compare them and they're different
Summary:
@public
Currently, every time we call into the packager we have to change the ulimit to make sure
we don't hit the EMFILE error (the packager uses as much concurrency as possible).
Using graceful-fs, the fs module -- with monkey patching -- becomes intelligent enough to recover
from EMFILE errors.
Test Plan:
* set `ulimit -n 256*
* start server
* request from your browser: http://localhost:8081/RKJSModules/MainBundle/CatalystBundle.includeRequire.bundle
* it works
Summary:
@public
Fixes [#393](https://github.com/facebook/react-native/issues/393). Currently the transformer assumes line-preserving compilers and defaults to a super-fast source map generation process. However, we need to support compilers that aren't preserving lines.
I had feared this wuold slow down the server but I came about a little known peace of the spec that defines an "indexed source map" just for the purpose of concating files: https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit
Test Plan:
1. runJestTests.sh
2. run server and click around example apps
3. add a custom transporter like babel
4. add a custom file and a debugger statement
5. debug in chrome and make sure it works
redbox still works
Summary:
cc @amasad
An error occurred while trying to display the Red Box since loadSourceMap was not included in the JS
bundle. This is because node-haste was treating its docblock as a multiline directive which doesn't make sense for `@providesModule`.
In loadSourceMap.js's case, the directive's value was parsed as "loadSourceMap -- disabled flow due to mysterious validation errors --".
There are two fixes: add a newline under the `@providesModule` directive, and change the module ID code to look at only the first token of the directive. I opted for the latter so we avoid this class of bugs entirely and AFAIK it's nonsensical to have multiple `@providesModule` values anyway.
Closes https://github.com/facebook/react-native/pull/866
Github Author: James Ide <ide@jameside.com>
Test Plan: Run the packager, trigger an error in an app, see the red box now show up again.
Summary:
@wez Mentioned this in Issue #239 -- right now when watchman takes too long we recommend you run `watchman` from your terminal which actually expects some arguments, so it prints out the following:
```
[pcottle:~/Desktop/react-native:changeErrorMessage]$ watchman
{
"error": "invalid command (expected an array with some elements!)",
"cli_validated": true,
"version": "3.0.0"
}
```
basically this ends up being more confusing since the command we recommend you run errors out, so lets change it to `watchman version` which at least exists cleanly.
I kept the troubleshooting link as https://facebook.github.io/watchman/docs/troubleshooting.html since it sounds like we will update that with the issue people run into in #239
Closes https://github.com/facebook/react-native/pull/825
Github Author: Peter Cottle <pcottle@fb.com>
Test Plan: Imported from GitHub, without a `Test Plan:` line.
Summary:
This PR teaches packager's `DependencyGraph` how to extract dependencies written with ES6 `import` statements.
It fixes the issue where you are not able to write your app with ES6 `import` statements when your custom transformer (replacing the default [JSTransform](https://github.com/facebook/jstransform), for example, [babel](http://babeljs.io/)) already supports the ES6 `import` syntax.
It will also be useful for [JSTransform](https://github.com/facebook/jstransform) later on once it implements `import` feature too.
Closes https://github.com/facebook/react-native/pull/386
Github Author: Pilwon Huh <pilwon@gmail.com>
Test Plan: runJestTests.sh
Summary:
See #406
Made sure the jest tests pass but didn't know a good unit test to add for this.
Closes https://github.com/facebook/react-native/pull/427
Github Author: Jacob Gable <jacob.gable@gmail.com>
Test Plan:
* ./runJestTests
* start app and click around
Summary:
Exposes the dev option that is already there to the CLI so that you can turn off invariant checks, etc. I also made it omit the inlined source map when dev=false which made it a lot faster to run on a phone, both due to smaller download size and fewer bytes to copy from Obj-C to JS and evaluate.
Closes https://github.com/facebook/react-native/pull/112
Github Author: James Ide <ide@jameside.com>
Test Plan:
* ./runJestTests.sh
* test bundle creation with `bundle.sh`
* test `load_dependencies.js` script
* start the server and click around shell app
Natansh Verma
Amjad Masad
Martin Konicek
Krzysztof Magiera
Martín Bigio
Christopher Chedeau
Alex Kotliarskyi
Ben Alpert
Felix Oghină
Felix Oghină
Evan Solomon
Thomas Aylott
James Ide
Dave Sibiski
Jing Chen
Forbes Lindesay
Bill Fisher
Tadeu Zagallo
Spencer Ahrens
Eric Vicenti
Nick Lockwood
Dmitry Soshnikov
Tim Yung
Kevin Gozali
Philipp von Weitershausen
Peter Cottle
Pilwon Huh
Jacob Gable
daviskoh
Martin Kosiba
Chengyin Liu