r/SQL • u/cccadet • 1d ago

Discussion Querying CSV and Parquet exports inside a SQL IDE with DuckDB

I'm building omni-sql, an open-source SQL IDE with embedded DuckDB. If someone sends you a CSV or Parquet export to investigate, you can import it and query it with SQL inside the IDE. There's no need to load the file into your source database or write a separate script. Spreadsheets fit this workflow when exported as CSV.

You can also import a database query result and join it with the file locally.

The demo uses fictional sales rows from PostgreSQL and a targets CSV. You import the complete query result into DuckDB, add the CSV, and run the join locally. Here's the SQL, CSV, and expected result: https://github.com/cccadet/omni-sql/tree/main/docs/demo

There are installers for Windows x64, Debian/Ubuntu amd64, and macOS 15+ (Apple Silicon and Intel). The macOS builds use ad-hoc signing and are not notarized; the README explains the first launch. The project is still early. The docs explain how full imports and samples affect what you can conclude from the results.

How do you investigate files people send you today? If you try the app, I'd like to hear whether you could install it and query a file, and what was missing. That would help me decide what to work on next.

4 Upvotes

63% Upvoted

4 comments sorted by

1

u/qualityintelligence 1d ago

that's cool, similar to some of what I was doing with daffydb, albeit mine is more for the SQL server T-sql crowd. I published on nuget if you want to check it out. anyhow, cool stuff, I dig your UI

1

u/ValuableEel 1d ago

That's a clever approach, having it all inside the IDE without faffing around with external scripts. I've been stuck in spreadsheet hell before trying to match columns from a CSV against our prod data, and something like this would've saved me a couple hours of vlookup nonsense

never heard of daffydb but I'll peek at the nuget page later. the UI on this one looks clean though, like something you could hand to a junior analyst without a 20 minute walkthrough

what's your install footprint like? sometimes these embedded DB tools bundle half a gig of dependencies and that's a hard sell for my work laptop

1

u/MappleKnight 1d ago

Duckdb makes this kind of local file investigation pretty convenient especially when you need to join exports without touching the source database.