Elixir with Flow is more than just parallel processing, it permits to break up complex flow in composable steps, to catch and handle error, finally to combine flows with RabbitMQ.
Flow (https://github.com/plataformatec/flow) is only a series of tubes. In Elixir everything can be expressed as a series of pipelines. The transformation of the data can be represented as many pipelines.
Elixir Flow allows to express the computation on collections with pipeline operator. It uses to allowing computation in parallel. There is places to use Flow, you should not rewrite all your code with this module, it's probably a bad idea. Flow is probably overkill for lot of purpose.
Counting words in a file, in the classic way, we open a stream, we create a map of the words and we convert it to list. If we use this method with classical Enum (https://hexdocs.pm/elixir/Enum.html), we will use lot of time. In this case, it is better to use the Stream (https://hexdocs.pm/elixir/Stream.html) library for the mapping. With Flow, the same pipeline can be executed in parallel.
Flow make your code maintainable by allowing composition of flows. There is two modes of operation, one historical (prepare, import and process all data) or in real-time (monitor source, run incoming data though the whole pipeline).
To compose flows, the good way is to create a module and create a flow function in it. It could be a Map or Stream, but it is usually a stream. It is possible to subscribe to a process, with from_stages (https://hexdocs.pm/flow/Flow.html#from_stages/2) function for example.
Based on different type of data, it is possible to branch and use different strategies. The simple case is to keep the branches inside the same flow but in certain condition, it is possible to create GenStage (https://hexdocs.pm/gen_stage/0.14.0/GenStage.html) consumer and deal with it differently.
Two sides of failure tolerance, with error handling and supervision. Starting a flow in a different process, and a supervision can be made as usual.
Let it crash philosophy can be used in Flow and it maybe the best way for many project.
Another method is flag and filter, if you want to have a flow running in long time, you need to use this method. prepare and filter functions should catch different and deal with them. The implementation is simple. This method has many benefits, minimal impat on the pipeline, the failure handling make things easier to see, and reduce stepping.
A more complex approach is to use branching errors, this method remove the expressiveness of the pipeline. It is a mix of Flag and Filter and Branching flow methods. It is only advised for complex error handling situations and when the number of errors is in the same ballpark as the success values.
It is possible to use simple scheduling with Flow by configuring them with start_prepare_flow function but if we want real-time processing, we need something different. It is also possible to schedule flows with RabbitMQ. A consumer is the entry of the flow and a producer is the output of the flow, it is a good way to branch things.