Background of command separation between manifest and collect

[tsteckenborn]

The way I understood it manifest gathers the files containing an fbt translation, while collect collects the individual strings and their description.

Could one explain the reasoning behind having them split? Doesn't that lead to the risk of just running one of them and receiving only partial results (e.g. no new strings or some files being missed)?

[cpojer]

I agree the cli needs a major overhaul before 1.0, and I believe we should merge the manifest and collect commands so that we have one command to collect and one to apply translations. We should also make collect parallelize across worker threads so that it's a lot faster.

I assume they are separate because at FB other tools might provide the manifest information. @kayhadrin may be able to provide more info on the background.

[kayhadrin]

I also think it'd be more convenient to have a default CLI command to do manifest+collect in one run.

Having said that, I still believe that the manifest and collect commands should remain public because some users will need them as part of their own custom build process (especially when they have a large repo scenario).

More background to explain the overall string collection process:

As you probably noticed, fbtee manifest needs to run first so that we have a complete awareness of all the available *$FbtEnum.js modules in the source code. They are needed so that we can figure out what strings come from the $FbtEnum module when we encounter <fbt:enum enum-range={My$FbtEnum}>. (More info: Shared enums)

After that, we can indeed run fbtee collect in parallel. At FB, it just so happens that that parallelization was done in a Hack script which would roughly work as follows:

1. Run fbtee manifest to generate .src_manifest.json & .source_strings.json
   🚀 Performance here hasn't been an issue even at FB scale.
2. Split .source_strings.json into chunks like .source_strings_chunk_n.json
3. Use parallel to collect strings from these JS files:
   ls .source_strings_chunk* | parallel --pipe fbtee collect ... > .source_strings_chunk_n.json
   🚀 Greatly improves collection speed for large repos since the Babel transform becomes a bottle neck.
4. Send extracted source string chunks to a "translation platform" (E.g. Crowdin)
   For convenience, we could have merged every source strings chunk (.source_strings_chunk_n.json) file to a single one. (Be careful since childParentMappings depends on an indexed array.)
   However, that's probably not really necessary since I suspect that most translation platforms can handle receiving source strings from multiple bundles.

Eventually, users can retrieve all their translations (keyed by fbt hashes) which can be consolidated into a single translation_input.json file. (This data structure is unordered so no worries there).

In conclusion, making fbtee collect parallel-friendly could be done but I don't think it's that important since the next step ("publishing the source strings") is essentially asynchronous.
You could also argue that the Unix philosophy "a process should do one thing and do it well" should apply in this scenario. :-)

|---- Legend ------|
| M: manifest      |
| C: collect       |
| S: send strings  |
| w: wait          |
|------------------|

Parallel collect, but wait for all before send:
M ->---- C#1 --->ww|             
   |---- C#2 -->www| S -------------> End
   |---- C#3 ----->| 

Parallel collect & send should be faster:
M ->---- C#1 ---> S#1 ---> End
   |---- C#2 --> S#2 ------> End
   |---- C#3 -----> S#3 ----> End

[cpojer]

With 1.2.0, manifest was removed. Everything is handled in collect now.