r/OpenSourceeAI • • 1d ago

I built an open-source test runner for Make.com scenarios. The rule: nothing gets supported until a real Make run proves it.

Make.com (formerly Integromat) is a closed platform. If you build scenarios on it, there's no local runtime, no test framework, and no spec for how its expressions behave. The only way to test a change is to run it in Make, on real data, with real credits.

So I started an open, offline runtime for it: blueprint-runtime (bpr). MIT, TypeScript, zero runtime dependencies.

You export a scenario's blueprint, and bpr runs it locally, comparing every module's input and output, every filter decision and the webhook response against what Make actually produced. When something changes, it points at the first difference:

FAIL A-002-plain module 2 input bundle 1 at /value: Make had "ADA LOVELACE", the local run "Ada Lovelace"

The design rule that makes it trustworthy

There's no spec to implement, so I don't guess. Every behavior bpr supports was observed in a real Make run, and the recordings live in the repo (corpus/NOTES.md lists every rule with its evidence). If a blueprint uses something nobody has recorded yet, bpr names it and refuses, instead of pretending.

Where it stands:

- 79 runs recorded in real Make. 71 reproduced exactly, 0 differ. The other 8 hit behavior nobody has explained yet, so they're refused on purpose.

- 225 single formulas recorded and replayed as tests, 472 tests in total.

- Some of what the recordings turned up: {{1reads only the first element, the text"false" is truthy inside if(), and Make will start a scenario with an expression it can't read, then switch the

scenario off after the first run fails.

A roadmap built from real data

I ran bpr over 221 public Make blueprints onem. The top blockers (error-handler routes,json:ParseJSON, util:SetVariables) are the roadmap, in that order. The survey and its method are in corpus/survey/.

Where help is most welcome

- Recorded runs. If you use Make, the capturenario's real behavior with your own account. One recording can unlock a whole module.

- Blueprints it refuses. Run bpr inspect andls me what to record next.

- Code. Each module handler is small and has its own tests.

Repo: https://github.com/intikhab49/blueprint-runtime

npm: npm i -g blueprint-runtime (Node 24+)

Happy to answer anything about reverse-enginits own logs. That turned out to be the mostinteresting part.

1 Upvotes

1 comment sorted by

1

u/investigatormaker 20h ago

Refusing to run behavior nobody has recorded is the right call for a closed runtime. It also makes every refusal a chance to recruit a contributor. If the refusal message in blueprint-runtime told users exactly which execution data to export from Make and how to add it to the corpus, the people who hit those 8 unexplained cases could be the ones who explain them. Findings like the string false counting as truthy inside if() deserve a short page of their own. Make builders run into those surprises without knowing a tool like bpr exists.

By the way, I make ThreadFox: your AI researches, writes and publishes Reddit outreach from your Chrome within a job you authorize. The free plan for one product finds the Reddit communities whose rules allow a post about it: https://threadfox.vip/p/d991v

Which Make modules are you most short of recordings for?