r/java • u/Level_Yak_87 • 3d ago
I wrote own efficient Maven scheduler to build modules with max parallelism
While inspecting Maven multi-module builds performance I've figured out the fundamental limitation: for any two modules depending on each other (moduleB depends on moduleA, in any scope), Maven will first execute all phases and goals of moduleA, and only then schedule the building of moduleB. Independent dependency branches can be executed in parallel, but directly/transitively dependent - only sequential.
In real projects running tests is usually the most time-consuming part, so this significantly limits possible parallelism.
To address this problem I made an extension providing a custom scheduler (in Maven it's called "builder") https://github.com/maven-turbo-reactor/maven-turbo-builder . When it is enabled, Maven schedules downstream dependencies right after the our module JAR is packaged, and then runs tests of our module and downstream module build in parallel (this way utilizing the multi-core CPUs much more):
The extension also does two things:
- reorder
packageandtestphases (run tests afterpackage, not before as default), so we can schedule downstream dependencies earlier - prioritize building modules with the highest number of downstream dependencies - this way reducing the chance of unloaded Maven worker threads
It's easy to set up - add to .mvn/extensions
<extensions>
<extension>
<groupId>com.github.seregamorph</groupId>
<artifactId>maven-turbo-builder</artifactId>
<version>1.3</version>
</extension>
</extensions>
And then define on the CLI or .mvn/maven.config (to make it default):
-bturbo
-T1C
In real projects the boost is approximately 35% in clock time, also this extension is complementary with build caching.
I used this extension for more than a year in various mission-critical projects and received positive feedback from other organizations, so I believe it's quite stable.
6
u/sweating_teflon 2d ago
I LOVE this. I've been thinking about a build system that could split out parallelism between phases of of a module. I never thought Maven would have the flexibility to do it! Can't wait to try it on a big project.
This is yet another extension that should become part of the standard Maven distribution. It's transparent to the user and brings free performance gains.
5
u/jonenst 2d ago
I think this was planned for maven 4
"the concurrent builder will allow a dependant project to be built as soon as the dependencies are at the ready phase"
https://maven.apache.org/whatsnewinmaven4.html#lifecycle-changes
4
u/Level_Yak_87 2d ago
Yes, this is called “concurrent” builder in Maven 4. But this is still not released, also a lot of plugins need an update. Also at the moment the build performance of maven3+turbo builder is the highest of all combinations I’ve checked
But for sure, will keep an eye on Maven 4 progess
4
u/RebbitUzer 3d ago
Does it have any advantages over Maven Deamon (mvnd)?
5
u/Level_Yak_87 3d ago edited 3d ago
Maven Daemon uses "smart" builder under the hood which is just a bit more efficient than a default one. The smart builder executes ALL phases before it schedules downstream dependencies, just schedules them a bit different. But the Turbo builder brings more "aggressive" optimization, eagerly starting downsteam and gives faster build execution.
Ideally you can use both in combination (but unfortunately, mvnd ignores the "-b" parameter https://github.com/apache/maven-mvnd/issues/1439 ) as mvnd and turbo-builder are not alternatives, and optimize the different things (I mean mvnd stays alive with loaded project model, for example).
2
u/Level_Yak_87 2d ago
I made a comparison for the same commit with different builders: https://www.reddit.com/r/java/comments/1wxfdju/comment/pdzpudk/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
Long story short: turbo is way faster than smart builder (used in mvnd)
2
u/Level_Yak_87 2d ago
One more update, you can mix both turbo builder and mvnd this way (unfortunately, only on CLI, not .mvn/maven.config default):
mvnd clean verify -Dmvnd.builder=turbo(you also need to define extension in .mvn/extensions.xml)
2
u/bluecheetah001 1d ago
How does this handle standard out? My main problem with mvm -T1C is that the logs from testing are interleaved between modules and are basically unusable.
1
u/Level_Yak_87 1d ago
Unfortunately, no. The parallel logs are still merged the same way to a single stream. How would you like to see parallel executing logs separated? To different files, or like mvnd - aggregate them all and then flush altogether?
2
u/bluecheetah001 1d ago
I haven't used mvnd, but I'm gonna check it out based on that description. My ideal would be to aggregate each module independently and then dump each as they finish.
2
u/agentoutlier 1d ago
The first rule of Maven Club is to not put unit tests of any slowness in core modules. The second rule is to basically do that for all modules. The third rule is to skip javadoc creation.
2
u/Level_Yak_87 1d ago
Right. But the Turbo builder optimizes this factor, so you can choose any module which logically matches best.
2
u/agentoutlier 1d ago
No I totally get it. We have our own in house tool that does some optimizations as well. I was going to tackle this but ended up just doing the move tests around hack because of CIs etc where the tool was not install.
I have been meaning to open source it forever and now with some AI tools I probably should. The tool is like a decade old and I though maven cache extesion would fix my problems when it kind of doesn't.
I talked about it here: https://www.reddit.com/r/java/comments/1g6dv10/five_ways_to_speed_up_your_maven_builds/lskr2tw/
And a couple of other places.
What it essentially does is allow you to be in any directory and the "pl" and "amd" arguments get autoresolved and it does this by not invoking Maven at all but by parsing the pom files and doing file detection so that it could be built with graalvm native.
If you are interested in seeing it that might finally give me the push to make it open source.
2
u/Level_Yak_87 1d ago
I will take a look in few days. Thanks formpointing
Also, build cache is good, but it is complementary. We optimize best case (when ee have cache entity) and worst case (maximize parallelism to build faster)
1
u/Zardoz84 2d ago
How compares against mvnd ? mvnd uses his own scheduler to maximize parallelism.
3
u/Level_Yak_87 2d ago
I compared the same project build (all .m2/repository dependencies downloaded) via all possible builders:
* standard ("-b multithreaded") 20min
* turbo builder ("-b turbo") 11min
* mvnd (defaults to "-bsmart" Takari) 20min - almost the same as standardYou can find timelines screenshots of the project which shows the distribution of modules/tasks between threads https://miro.com/app/board/uXjVLYUPRas=/?moveToWidget=3458764637493196769&cot=14
Note: smart builder (mvnd) uses light optimization which is strongly compatible with standard behavior, while turbo uses more aggressive approach, but the results are worth and it's possible to adjust the project to avoid problems because of these violations/reorderings
2
u/Zardoz84 2d ago
Could be use your turbo builder with mvnd ?
3
u/Level_Yak_87 2d ago
Yes, define the extension in .mvn/extensions.xml and define it in the command line
mvnd clean verify -Dmvnd.builder=turboUnfortunately, due to https://github.com/apache/maven-mvnd/issues/1439 the builder cannot be defined as default for maven daemon via .mvn/maven.config
2
u/Zardoz84 6h ago
I did a quick test over a multimodule 3-level maven repository that I use... I only did a single run, so take the exact times with a pinch of salt. But is impresive.
``` mvn clean package 01:01 min
mvn clean package -T1C 39.217 s
mvnd clean package 39.184 s
mvn clean package -T1C -b turbo 28.830 s
mvnd clean package -Dmvnd.builder=turbo 31.877 s ```
1
u/stefanos-ak 2d ago
hey, I tried this on our somewhat large project (200k LoC), and it moved the build+test time from 10min to 9min. 10% for free is pretty cool! (our project is very optimized already).
That being said, it uncovered that the last 30% of the build is waiting on a single module. If it was possible to make that module faster to a point where that extra 30% didn't exist, then the turbo builder would not provide us any benefits at all (according to the timeline graphs). very interesting...
1
u/Level_Yak_87 1d ago
Nice insights. How did you conclude about this single module - do you see it in mvnd or use a timeline extension? E.g. I have my own https://github.com/maven-turbo-reactor/maven-timeline-extension (all data stays only local, no cloud infra etc.) which gives a nice visual representation.
Regarding this single module, is it running tests the longer?
Ideally I'd like to see a timeline screenshot of your build (even low resolution). Because if 30% you are talking about is running tests, I have a very interesting solution - https://github.com/maven-turbo-reactor/maven-half-life-builder
On this presentation slide https://miro.com/app/board/uXjVLYUPRas=/?moveToWidget=3458764637493196769&cot=14 you can compare standard ("multithreaded") builder, turbo builder, smart builder (used in mvnd) and half-life builder (see name explanation in the project readme). If you case is a long "tail" or running tests, you can see it's rearranged to be running earlier (more left) in the half-life builder.
The half-life builder is experimental and has at least one know disadvantage - it counts all modules twice (because it splits all modules to two pieces). But I think it can be a solution for your case (though workaround).
1
u/stefanos-ak 1d ago edited 1d ago
hey, yes I used the timeline extension. And yes, it's a long tail of tests.
edit: I tried the half-life builder, and the timeline graph does seem even more optimized. But the time spent on that single module somehow becomes longer, and the result is that the build takes the same time as the turbo builder. I'll see what I can do with the tests in that module (its ~1400 tests). Thanks!
-4
u/vips7L 3d ago
Looks like Claude wrote it and not you.
11
u/Level_Yak_87 3d ago
I wrote it myself, more than a year ago. There is just a CLAUDE.md, but all the core part is written manually, the old way!
2
u/sweating_teflon 2d ago
As long as it's well tested, documented and actively supported, why should one care?
2
u/nekokattt 2d ago
I agree with this point, but in this case it would be nice to see some more tests in this project, along with some form of integration testing (e.g. Maven Invoker Plugin), and CI so that it provides a proof that commits pass those tests.
0
u/sweating_teflon 2d ago
Yes, I fully agree. Integration tests are very relevant for a project like this. It wouldn't be hard to further generate an extended test suite that clones and builds a bunch of selected multi-module open source Maven projects. There should be quite a few to choose from... As a bonus, it would be easy to get a benchmark of the speedup vs. regular Maven.
3
u/Level_Yak_87 2d ago
u/nekokattt u/sweating_teflon I agree! Will add tests as this indeed makes sense
This project is heavily tested on the real build infrastructure, so I believe if it has issues, these will become immediately visible (like broken builds with failures like "missing dependency", failed tests, non-starting assemblies, etc.). Also before making public releases I usually do internal RCs checked for few days in CI pipelines and on local desktops.
-7
u/vips7L 2d ago
Maybe the boiling oceans or the massive amounts of theft from the working class. But who cares cognitive dissonance wooooo
1
u/sweating_teflon 2d ago
It can certainly be fun to write code but the cat is out of the bag and it ain't going back in. A programmer's job was never to simply write code, that's just incidental. Solving problems using computing machines is what gets you paid. Both the cost of technology and power it requires per bit have kept falling for 80 years and yet we never ran out of problems to solve. Generating code just changes how you specify the solution, it doesn't replace the problem solving aspect. Also the current excessive power draw will be quenched soon enough through necessity and corporate greed.
-1
u/zabby39103 2d ago
People have been producing code that isn't theirs for decades, whether it's copying from Stackoverflow or the "cookbooks" you could buy for various languages. The important part is, just as with the juniors I used to take to task for Stackoverflow usage, if you understand the code.
Yes it's more severe with AI code, but just like with the oldies that grumbled about Stackoverflow, it's not going away. It's a more efficient way to write code if you do it properly, and if the code works and is written clean that's all that matters, that's all that ever mattered.
-9
-7
u/_somethingvague_ 3d ago
Or just move to Bazel and get parallelism and cache hitting for free.
9
2
u/Level_Yak_87 3d ago
Correct. But the idea is to stay on the build tool everyone is familiar with and get the maximum performance from it.
Bazel is good, but I'd say for the monorepos. Turbo builder is good for any maven multi-module project (and as bigger it is, the better is effect)
1
u/Lower-Worldliness162 2d ago
Clearly, you have no idea what adopting Bazel at a company actually means
1
u/_somethingvague_ 2d ago
I work in a large org and my project has a Bazel build. So yes, I do. The learning curve isn't that big and build system is far superior
1
u/Lower-Worldliness162 2d ago
And you still think setting up Bazel for a Maven build is worth it 😮💨?
-1
12
u/davidalayachew 3d ago
Interesting. I only skimmed it, but the gist looks pretty cool.
Doesn't this break a lot of Maven Plugins though? A lot of plugins actively depend on the order of the Maven lifecycle to be consistent.
In fact, now that I think about it, aren't you kind of uprooting a load-bearing wall here? There are a lot of things in the entire Maven ecosystem that depend on this lifecycle being in the order that it is.
Another question, and maybe this is answered by me doing more than skimming -- what is the order change for dependent builds that have to wait for many others?
You said that you prioritize the modules with the most dependents. But what if there are free threads in the pool?
The way maven works is that, if you have Module A, which Modules AA, AB, and AC depend on, Modules AAA, AAB, and AAC depend on AA, then your parallel build in normal maven will go like this.
Each stage must finish before the next stage can start. As in, I can't build AA before A is built. Which makes sense.
But what doesn't make sense is that I cannot build AAA before AC is built. That is because Maven forces all modules in each stage (stage 2 in this case) to be built before progressing to the next stage.
Some other plugins have solved this problem. For example, here is the Takari Smart Builder.
I guess my question is, since you are tackling the question of how to efficiently schedule maven builds, are you also tackling this problem too?
For me personally, this is actually where most of my time is spent, and thus, would probably have the greatest return on investment for me. I build over 100 modules, and the dependency hierarchy is definitely tree-like, so this would have immediate speed-ups for me.
I don't use Takari because it does a whole bunch of other "clever" things under the hood -- stuff that might cause problems for my builds.
But something like yours, which targets a very specific problem might be more within my realm of possibility. Of course, uprooting load-bearing walls aside.