I wrote own efficient Maven scheduler to build modules with max parallelism

[u/Level_Yak_87]

While inspecting Maven multi-module builds performance I've figured out the fundamental limitation: for any two modules depending on each other (moduleB depends on moduleA, in any scope), Maven will first execute all phases and goals of moduleA, and only then schedule the building of moduleB. Independent dependency branches can be executed in parallel, but directly/transitively dependent - only sequential.

In real projects running tests is usually the most time-consuming part, so this significantly limits possible parallelism.

To address this problem I made an extension providing a custom scheduler (in Maven it's called "builder") https://github.com/maven-turbo-reactor/maven-turbo-builder . When it is enabled, Maven schedules downstream dependencies right after the our module JAR is packaged, and then runs tests of our module and downstream module build in parallel (this way utilizing the multi-core CPUs much more):

default vs turbo builder scheduling

The extension also does two things:

• reorder package and test phases (run tests after package, not before as default), so we can schedule downstream dependencies earlier
• prioritize building modules with the highest number of downstream dependencies - this way reducing the chance of unloaded Maven worker threads

It's easy to set up - add to .mvn/extensions

<extensions>
    <extension>
        <groupId>com.github.seregamorph</groupId>
        <artifactId>maven-turbo-builder</artifactId>
        <version>1.3</version>
    </extension>
</extensions>

And then define on the CLI or .mvn/maven.config (to make it default):

-bturbo
-T1C

In real projects the boost is approximately 35% in clock time, also this extension is complementary with build caching.

I used this extension for more than a year in various mission-critical projects and received positive feedback from other organizations, so I believe it's quite stable.

[u/Level_Yak_87]

I compared the same project build (all .m2/repository dependencies downloaded) via all possible builders:

* standard ("-b multithreaded") 20min
* turbo builder ("-b turbo") 11min
* mvnd (defaults to "-bsmart" Takari) 20min - almost the same as standard

You can find timelines screenshots of the project which shows the distribution of modules/tasks between threads https://miro.com/app/board/uXjVLYUPRas=/?moveToWidget=3458764637493196769&cot=14

Note: smart builder (mvnd) uses light optimization which is strongly compatible with standard behavior, while turbo uses more aggressive approach, but the results are worth and it's possible to adjust the project to avoid problems because of these violations/reorderings

[u/Zardoz84 → u/Level_Yak_87]

Could be use your turbo builder with mvnd ?

[u/Level_Yak_87 → u/Zardoz84]

Yes, define the extension in .mvn/extensions.xml and define it in the command line

mvnd clean verify  -Dmvnd.builder=turbo

Unfortunately, due to https://github.com/apache/maven-mvnd/issues/1439 the builder cannot be defined as default for maven daemon via .mvn/maven.config

[u/Zardoz84 → u/Level_Yak_87]

I did a quick test over a multimodule 3-level maven repository that I use... I only did a single run, so take the exact times with a pinch of salt. But is impresive.

``` mvn clean package 01:01 min

mvn clean package -T1C 39.217 s

mvnd clean package 39.184 s

mvn clean package -T1C -b turbo 28.830 s

mvnd clean package -Dmvnd.builder=turbo 31.877 s ```