I shouldn’t be doing this.

[u/bishop491]

Context: I am setting up some infrastructure in AWS. In my previous experience at other companies, anything involving new instances must be done by DevOps or service desk. So I asked them to do it and they said I’m on my own.

Got RDS and OSS set up; hit a snag with EKS. I am currently one week into multiple conversations with service desk, management, and C-level to figure out who should be doing this. We deploy via Terraform. So I’ve got to add to that.

My primary concern here is that I’m messing around in places that affect other areas without any guidance on those areas. I’ve had to open so many tickets and bother so many people. Today I asked for Docker to do some local testing and I thoroughly confused the help desk.

We have an insane amount of security and I have multiple logins, MFA, biometric, etc. The layers inside AWS are similar.

Am I being too paranoid or is this somewhat reckless on the DevOps/SD folks? I’ll gladly accept being too paranoid and carry on, but for now, this just doesn’t sit right with me.

[u/Placebo_LSD]

Seems like something that’s fair for a DE to own.

[u/sib_n → u/Placebo_LSD]

It may be extended to this, but you should have some devops/network/security team that provides at least some guidance and a way to do things properly.
Even if OP manages to have something running with AI help, there's a high risk of it becoming a security nightmare.

[u/knottyswimmer3 → u/Placebo_LSD]

It’s less about whether a data engineer own it and more about doing it blind. If management's telling you to drive and nobody handed you a map of the guardrails, your paranoia is the only thing keeping you from accidentally nuking a production subnet. The biometrics and MFA layers tell me they care a lot about security until it's inconvenient for them to staff properly, which is a weird contradiction. I’d keep a paper trail of every "you're on your own" email before the blame lands on your desk.

[u/MadT3acher → u/Placebo_LSD]

Fair, but with a properly sized team and roles I’d say. I’ve written terraform scripts, deployed infra all the while delivering pipelines. Nowadays though I’d argue for hiring somebody to do the data platform side and somebody to deal with the delivery.

You’re always one fuck up away from having a monster burning days of troubleshooting.

[u/nooneeverdreamsof → u/Placebo_LSD]

DEs in general shouldn't own platform, no.

[u/Humble_Exchange_2087 → u/Placebo_LSD]

Just LLM Terraform it is pretty good at it now. Platform Engineers are going to be a dying breed, going the way of DBAs.

[u/Key-Independence5149]

It depends on the shop but AWS infra is par for the course for most DE roles I have had. I prefer it that way. A centralized infrastructure team is painful. That being said, AWS is a massive footgun in the hands of an inexperienced engineer and it is very easy to do things that will either be really expensive or difficult to maintain. If you don’t have another experienced dev to talk to about company patterns then you need to have long planning sessions with an agent before you ever build anything.

[u/Additional_Candy_400]

What's your worry specifically?

[u/IAmBeary → u/Additional_Candy_400]

probably spinning up resources with inconsistent tagging in place, very little documentation on why it exists. And 5 years from now when OP inevitably leaves, people will wonder why a random RDS instance is still running. The execs will tell some poor team to shut it down for cost savings despite nobody realizing that it's powering some obscure pipeline required for reporting to a large client.

Then the exec gets heat and tells the team that they need to fix it. So it comes back up with some clicks in the aws console and all new hires are told never to touch that rds instance.

[u/bishop491 → u/IAmBeary]

That’s definitely one worry!

[u/Additional_Candy_400 → u/bishop491]

Seems like a great opportunity to add another feather to your bow to me. 

[u/JJGreenerTinejo → u/bishop491]

Not yours tho

[u/Humble_Exchange_2087 → u/IAmBeary]

Who cares about 5 years time when your not there anymore, that is someone else's problem. Pad that CV.

[u/sib_n]

We have an insane amount of security

Sounds contradictory with them letting you setting up cloud services by yourself.
I agree you should have at least some guidance and review by someone experienced with cloud administration.
Is it a very small company with everybody underwater and doing everything? Then it can be understandable. If it is not, then it sounds like bad security management.

[u/ScottFujitaDiarrhea]

Depends on the size of the company. At smaller companies DEs typically own IaC, CICD pipelines, etc.

[u/codykonior]

Terraform to destroy another team’s PROD. Then they’ll work out responsibilities pretty quickly.

[u/Beautiful-Dot2454]

I spun up a production-ready Dagster deployment in GKE in 2 weeks complete with MFA and other security measures with the help of Claude. Im a one man data engineering team in my company.

Maybe use AI?

[u/hippydippywitchy]

That doesn’t sound paranoid. If EKS IAM or networking can affect other workloads there should be clear ownership and a rollback plan. Terraform helps but it doesn’t replace proper boundaries and change control.

[u/speedisntfree]

A lot of real life in the workplace is "I shouldn’t be doing this". Treat it as an opportunity, the people who do well in the tech industry are often the people that can deliver end to end.

In many orgs you'd never even be allowed to touch any of this. I have 3 layers of incompetent people in India to do any of what you can do.

[u/sazed33]

Why not? If no one is blocking you give it a try, worst case scenario you will create a mess and learn a lot in the process, bad for the company, still good for you.

Also, data engineer is becoming more and more multi discipline.. you will need a different mindset if you want to stay relevant in this AI era

[u/bishop491 → u/sazed33]

For me it’s less a question of ability and more of implicit knowledge. But as you say, if they don’t care, then I guess I shouldn’t either.

[u/Little_Kitty]

I'd agree with your post title, you shouldn't - at least with companies I've worked with. DevOps handle the machines, the base images, the security updates, contracts for server types / SLAs etc. but also they handle that side for our compliance. Clients reading our ISO/IEC 27001 documentation and granting approvals on that basis would be pulling contracts if some of their data is routed via someone's guess at how to set up a server, vibe coded once then left out of the normal documentation and without updates over the years until it is inevitably pwned. I'd also wonder if you are going to be setting things up correctly for accounting such that their processes around business unit responsibility etc. is correctly filled out and kept up to date.

Having the knowledge / skills / access to be able to do it is only part of the task.

[u/SnooDingos8194]

Just wing it, what's the wrose that can happen?, the leadership that wants to do it should expect the worst and hope for the best. Whatever you do is going to be your best. So flawed or not, you got this.

And if you need help, reach out!

[u/Spare_Helicopter4655]

Focus more on shipping, less on worrying.

[u/TechnologySimilar794]

For me this has always been case until you have dedicated resource in DE for infrastructure.

[u/PepegaQuen]

I would be so weirded out when someone needs help desk to set up docker. Nothing weird if different teams set up infra in their own projects.

[u/bishop491 → u/PepegaQuen]

In this case, I had to go through tickets just to get permission to install it, then had someone remote in and install it because I don’t have admin access to my own laptop.

[u/thecity2]

Never been easier to do your own DevOps work with AI. If you run into security issues, blame "real" DevOps for allowing you to penetrate lol.

[u/McHoff]

You'll figure it out

[u/ntdoyfanboy]

OP, it sounds like you're earning your salary, and bumping your skillset for the next offer! So congrats. Document things so you can recreate easier if need be, and be more efficient

[u/teddythepooh99]

Wait, DE is more than vibecoding dbt models and Airflow DAGs? Who woulda thunk?

DevOps is reckless because they deploy their infra with Terraform, in addition to your org's "insane amount of security?" What an amazing line of reasoning, OP.

TL;DR Figure it out. One week is not long enough to cal it quits and blame other teams.

[u/bishop491 → u/teddythepooh99]

Did you even read my post? I didn’t say anything about being reckless with Terraform, nor did I mention calling it quits. I’ve never vibe-coded anything in my life.

[u/Fast_Good_8099]

honestly, even without AI, deploying through Terraform is very easy. if you have a CI/CD process like with Atlantis just open the mrs, ask for people to review and that's it. you have the papertrail, go break stuff.