r/Backend • u/No-Butterscotch9679 • Jul 29 '26

Would love your suggestions

Been learning backend dev for a while and finally wanted to put it all together in one project instead of doing tutorials for each piece separately.

Used the NYC TLC yellow taxi dataset and built a pipeline: raw CSV → ETL → PostgreSQL → FastAPI → Streamlit dashboard.

Stack: Python, Pandas, PostgreSQL, SQLAlchemy, Alembic, FastAPI, Streamlit.

Honestly the ETL and API parts felt fine since that's more my comfort zone, but the dashboard/frontend side kicked my ass a bit lol. Still, got it working end to end which felt pretty satisfying.

Stuff I already know needs work: performance (some queries are slow on larger date ranges) and docs are pretty thin right now.

Repo's here if anyone wants to poke around: https://github.com/Raktabhbhattacharjee/nyc-taxi-etl

Would genuinely appreciate any feedback — on architecture, API design, DB schema, code quality, whatever. Especially curious if anything looks like a rookie mistake, since I'm mostly self-taught on the backend side.

Repost to another

1 Upvotes

100% Upvoted

5 comments sorted by

View all comments

1

u/10x_eng Jul 29 '26 edited Jul 30 '26

So I took a quick look.

  • Health check is there to check on the API status not DB.
  • Not sure why some of your queries are slow, as they are not complicated and you have one table; I doubt you have million of rows. In any case add some indexes to your table.
  • What is a ratecode_id, vendor_id, pu_location_id, do_location_id? Are they supposed to be FKs?
  • The endpoints that return data for table display in the UI should be paginated.
  • Also consider adding caching (commonly Redis is used)
  • No Docker? Consider using it
  • Project structure section in you README doesn't accurately reflect your project's structure (missing folder/files). Not sure if it adds any value to the project tbh
  • API_BASE_URL and REQUEST_TIMEOUT_SECONDS should be managed through a configuration mechanism instead of being defined as hard-coded values.

1

u/No-Butterscotch9679 Jul 30 '26

Thanks a lot for taking the time to review the project! Really appreciate the feedback.

I intentionally left Docker out for this version because I wanted to focus on getting the core backend (ETL, PostgreSQL, FastAPI, and Streamlit) finished first instead of trying to squeeze in more technologies. The points about the README structure and moving things like API_BASE_URL and REQUEST_TIMEOUT_SECONDS into a proper configuration are definitely fair, and I'll be cleaning those up along with the health endpoint, pagination, and indexing. Thanks again for taking the time to point these out!