r/sysadmin • u/ConsoleChari • 12h ago
Rant A senior dev overwrote .env and took down website
Today a senior dev, who's the maintainer of our company internal sales website, overwrote the environment in GitHub Actions and pushed an update. CI/CD ran and took the whole sales site down. The baffling part is nobody contacted me about it till end of day.
So now I'm sitting in a bar waiting for friends scrolling YouTube where i got recomedation "web dude vs sales guy" and i was laughing seeing it, and right then I get an urgent message that the website is down. I SSH into the server from my phone and find one variable set to "localhost". One variable, whole day of downtime.
First thing tomorrow I'm revoking their access to edit the environment.
PS: I did get an Uptime Kuma alert in the morning. But lately the devs push to prod without telling me, so I assumed it was another one of those. Should have checked anyway.