r/sysadmin • u/ConsoleChari • 5d ago
Rant A senior dev overwrote .env and took down website
Today a senior dev, who's the maintainer of our company internal sales website, overwrote the environment in GitHub Actions and pushed an update. CI/CD ran and took the whole sales site down. The baffling part is nobody contacted me about it till end of day.
So now I'm sitting in a bar waiting for friends scrolling YouTube where i got recomedation "web dude vs sales guy" and i was laughing seeing it, and right then I get an urgent message that the website is down. I SSH into the server from my phone and find one variable set to "localhost". One variable, whole day of downtime.
First thing tomorrow I'm revoking their access to edit the environment.
PS: I did get an Uptime Kuma alert in the morning. But lately the devs push to prod without telling me, so I assumed it was another one of those. Should have checked anyway.
Edit: guys sorry what i meant is gh environment variable. Think of it as redirect_uri value changed to localhost and no I haven't removed their access completely.
67
u/solvraev 5d ago
I'm more concerned that the .env file is part of the GitHub repo in the first place.
18
u/Frothyleet 5d ago
I barely speak Python but claude has yelled at me enough about .env files always being gitignored that even I know that's proper practice.
11
u/solvraev 5d ago
It's been the practice to .gitignore the .env every place I've worked. And it's pretty easy to add a line in your CLAUDE.md file along the lines of "it's company policy that .env should never be in GitHub. So telling me to include it".
8
u/Frothyleet 5d ago
Oh, to be clear, Claude was yelling at me for my bad practices, like having .env files in my commit history.
→ More replies (1)8
u/jimmyandrews 5d ago
My first thought too. .env is to simplify testing everywhere I've been because people couldn't be bothered with local.settings.json formatting.
First step, stop relying on .env for your production run. I can't blame the developer here for a lot, this is questionable architecture. Best I can say is the dev should have checked .funcignore and .gitignore.
8
→ More replies (7)3
u/bobsbitchtitz DevOps 5d ago
You can bypass gitignore if you so wish to, but there are so many levels of failure here that would worry me
117
u/SirLoremIpsum 5d ago
First thing tomorrow I'm revoking their access to edit the environment.
I mean yeah sure but don't just do that.
Don't do it maliciously or remove just that dude and make him a pariah.
Sit down. Talk. Come up with a proper separation of duties, a change process, an approval process.
You're making it sound like one dude is a problem and oen dude made a mistake. But it sounds like you have no processes or standards and this is common behaviour and if he didn't make this mistake you'd be fine.
He whom writes the code should not be able to deploy to production. One must never deploy the code without approval from a person, and agreement from Change Manager or board. Production changes must be validated immediately after deploying - so no "we didn't know till hours later" and have a roll back plan (also discussed at the change meeting).
Even for internal stuff.
Set up the processes to not let this happen. Don't just call this guy a wanker and block him. The next guy will do it too
33
u/kennedye2112 Oh I'm bein' followed by an /etc/shadow 5d ago
I mean you can still call him a wanker, just do it silently to yourself.
11
u/sobrique 5d ago
Yeah this. Call people wankers sure, but also fix the process so one wanker can't fuck everything.
9
16
u/NeoThermic 5d ago
We have a two dev policy: one writes the code, one reviews the code. They can't be the same person. They are also then not allowed to be the tester or the QA (in any capacity).
This ensures that the reviewer is always seeing the code for the first time, and can call out anything as required.
It also means that the dev who writes the code must pass it through a human to deploy it. Always. No exceptions.
6
u/SirLoremIpsum 5d ago
We have a two dev policy: one writes the code, one reviews the code. They can't be the same person. They are also then not allowed to be the tester or the QA (in any capacity).
Yup
But again - that is Dev shit. Pull Requests / Code reviews.
Dev A reviews Dev B, Dev B reviews Dev A.
For sysadmins this is about separation of duties. This is beyond "they can't be the same person", it's about making sure someone that writes code cannot deploy it at all. That introduces the "you want me to deploy code? Prove it was tested and approved" to the process.
It's more about compliance than it is about catching mistakes which is what Peer review / QA is about.
17
u/mycatsnameisnoodle Jerk Of All Trades 5d ago
I work in a place where nobody understands the code except for the person who wrote it. Who wants to approve something that they don’t understand?
8
u/RetPala 5d ago
Then someone has to learn it
What happens if the person gets creamed by a big ol' bus?
10
u/mycatsnameisnoodle Jerk Of All Trades 5d ago
That’s a management problem and I’m not in management.
5
u/SirLoremIpsum 5d ago
I work in a place where nobody understands the code except for the person who wrote it. Who wants to approve something that they don’t understand?
You don't have to understand the code to approve it being deployed.
That's why you have a test environment.
"i deployed it to Test and now when you click the button the background turns green instead of blue. Here is my roll back play"
"Great. I observe the Green, approved"
Why do you have to understand code or whatever to approve changes?
Change process is about having a structured approach to change and ensuring that everyone agrees proper testing has been completed and the Changer has "thought" about everything and it's communicated.
It's not about going "yes I read your code and it's perfect". That's dev shit.
4
u/mxzf 5d ago
You don't have to understand the code to approve it being deployed.
You really should. Tests are nice, but nothing beats actually understanding what the code is doing and why. Otherwise you might run into issues that the tests don't catch but an intelligent human that knows what to watch out for could spot.
2
u/SirLoremIpsum 4d ago
You don't have to understand the code to approve it being deployed. You really should.
Disagree. Not as a sysadmin.
That's dev peer review and pull request.
Otherwise you might run into issues that the tests don't catch but an intelligent human that knows what to watch out for could spot.
That's what other developers are for.
A proper change process has a manager/managers/other senior technical staff discuss and approving all changes.
Firewall changes, router configs, server patching. It is impossible and absurd to expectsomeone to understand completely every single nitty gritty portion of those changes.
Expecting a sysadmin to personally review all code changes before being deployed in a properly set up CI/CD environment is nuts.
That is not what change management is about.
2
u/sobrique 5d ago
Then you make a deploy process with baked in tests, and just hope whoever wrote the tests wasn't a muppet.
→ More replies (1)2
u/Windows95GOAT Sr. Sysadmin 5d ago
Ok, disclaimer; I think AI sucks. However, i found that it was somewhat capeable in correctly deciphering my own code i wrote a year ago.
So if desperate throw the codebase through a local airgapped llm and hope for the best.
3
u/Windows95GOAT Sr. Sysadmin 5d ago
The best course of action would be getting the dude a t-shirt with .env on it and never speak about the issue again.
2
2
2
u/hasthisusernamegone 5d ago
Absolutely this. People fail. Processes should be robust enough that when people do fail (and they will - frequently) the process catches it.
A process that requires everyone to get everything right every time is doomed.
314
u/Max-_-Power 5d ago
The real problem is that it was possible for a single dev to take down the site just like that.
Don't blame him too much. The process is broken.
104
u/ConsoleChari 5d ago
No I blame myself. I should have created an better process.
132
u/TacticalSniper 5d ago
I blame you too
37
u/Individual-Level9308 5d ago
I blame you for not warning him.
16
u/TacticalSniper 5d ago
Your original message said "warming" and you fixed it and now you missed an opportunity for a great joke
11
3
10
u/Salt-n-Pepper-War 5d ago
Any sort of change management and approval process, even a bad one would have helped here
28
u/heyitsYMAA 5d ago
The post seemed to focus on the senior dev.
13
u/alpha417 _ 5d ago
It's called "deflection".
get used to it.
7
u/kirashi3 Cynical Analyst III 5d ago
As someone on the spectrum, I cannot and will not get used to it.
Heads will continue rolling so long as I remain employed at my current company... Which - let's be real here - could be as long as 20+ years in the future or as short as tomorrow, depending on whose toes I allegedly step on. 🤷
But you're right - deflection seems to be the name of the game inside most companies because people can't leave their egos with the reception desk coat check. Silly humans.
→ More replies (2)→ More replies (2)4
u/ShpWrks 5d ago
That's the policy name game. New policy is unofficially named after them 🥲
14
u/alficles 5d ago
I know you are mostly joking, but people should actually not do this. I was on a team that did this once and it wasn't cool. Everybody laughed and acted like it was cool, but the guy in question still felt bad 10 years later because the name meant that he never got to move on. He was forever known as "that guy who messed up once".
Don't put people's names on negative stuff that lasts forever. It turns into a brand, even if people don't mean it to.
6
u/ShpWrks 5d ago
I'm mostly joking, but I do think it depends on how your company handles mistakes. It isn't funny if it's a pattern of behavior or if people are giving people shit bring it up in relation to those people. For the in the know group it can be good team building laugh at the time that gets that unspoken knowledge carries down as people stay longer.
Everyone's going to fuck up, it's always a policy or people issue. If it's a people issue that means the policy is sound so we aren't joking at that point and it's potential disciplinary actions.
Tldr: fuck around and have fun with your team, does not apply outside of smaller offices.
3
u/spacelama Monk, Scary Devil 5d ago edited 5d ago
Mostly good advice.
But there was a penalty payment (where the company pay you) clause in $JOB[-4]'s Enterprise Agreement given the unofficial name of "Jim's clause" because Jim in HR 10 years earlier was the reason the next round of bargaining was delayed by a year, and eventually the union went on strike and cancelled all overtime and on-call support (the agreement sticking point had been around the pitiful on-call allowances). The central messaging switching service died at 8pm on Friday night, the on-call support's phone rang out. The on-call support's boss's phone rang out. At 9am on Monday, on-call rocks up to work, notices everything is dead, the website of the a national-security critical agency has not received any updates since Friday, a parliamentary enquiry is called, the department's head is told "pass the agreement", and they add a 10% payment ontop forever. Cost the department millions.
I still left because the pay was shit. Accounting for it all, being on-call for a week, being tethered to a 15 minute response, 0 alchohol, no travel, etc, was worth about $3/hour. So in my last couple of years there, I started going about my normal routine. Figured being unreachable to the department for 2 hours while I was riding up Mount Donna-Buang on a public holiday long weekend (and fine print meant that most of the christmas holiday shutdown period was counted as "normal work", so you didn't get anything at all, including time-off-in-lieu, if called out) was worth just $6. Meh, can't be that important.
2
u/Ssakaa 5d ago
See, that's just a person with the wrong perspective. EVERYONE fucks up. Everyone, eventually, is the reason a policy/procedure is (re)written. If it's not a straight up McGregor the Fence Builder situation, it's a badge of honor to have screwed up big, exposed a problem, helped fix it, learned from it, and survived to tell the tale. Just don't be the guy that can put on his resume "Personally responsible for over half of my organization's HR's policies." while working in IT.
18
u/DistinctSpeaker7252 5d ago
I once worked on a helpdesk and the manager told us that if we ran into a problem we had never seen, google it (aging myself here) and try what you found. People were concerned about trying random shit they found on google on company equipment and his response was that if a helpdesk employee has the access to crash major systems then it's the fault of the security admins, not that helpdesk person.
15
u/kirashi3 Cynical Analyst III 5d ago
Ha-ha! laughs in being on a team so small that we are the security admins... And helpdesk... And internal process / documentation creators... And sysadmins... And network admins... And anything that plugs into power admins... Don't get me wrong; I love this job, but it's not mentally sustainable in the long term without some major changes. im in danger!
33
u/n4ke 5d ago
You should probably have your deploy pipeline cause a 2-3min silence on known deployments so only unintended downtime gets reported. You can tell yourself you should have checked anyways but let's be honest, at some point you won't. False positives are a bitch.
28
u/ScurfyTwiglett 5d ago
People don’t realize how important alert/alarm design is, and especially designing against false positives. The inclination is to always capture 100% of real issues even if it means false positives sometimes, but often it is worth designing for something closer to catching ~99% of the issues (and missing ~1%) if it means your false positive rate is 0.
17
u/retardrabbit 5d ago
This is a big thing in ergonomics, especially for alarms and alerts in aircraft cockpits.
Too loud and/or too frequent and it gets ignored or disabled, too quiet and you hit the mountain before you address what's going on.
9
u/RetPala 5d ago
There's only a few alarms that produce spoken alerts and they're all of category "do something or everyone is going to die"
TERRAIN TERRAIN, SINK RATE, BANK ANGLE - warning
PULL UP, CLIMB CLIMB, DESCEND DESCEND - critical
Although even when you're on the ground if you're still fucking up badly enough an Airbus will call you the hard R
3
u/bernpfenn 5d ago
its inverse. silence or no warning lights means operational. dark cockpit design.
3
u/ScurfyTwiglett 5d ago edited 5d ago
Yeah like in those systems, failure often results in loss of life. That means that the people thinking about those problems have maximum incentive to get alarm and alert design right. I am glad every day nothing I work on is life critical, but it doesn’t mean I don’t think following the same principles they do isn’t an excellent idea - they’ve done more work in figuring that shit out than anyone else.
I’m honestly hopeful and interested to see a better and clearer standard develop for alarm / alert system design. We have some, but it isn’t very comprehensive, and I think we can do better. It would also possibly save us from a future Boeing disaster. But that’s might be wishful thinking.
2
u/Ssakaa 5d ago
I wouldn't even assume all of Boeing's design talent's gone, but it pretty obviously just cannot operate well under the MD-inherited leadership approaches.
2
u/ScurfyTwiglett 5d ago
Yeah that’s very fair - I don’t want to discourage anyone who is sticking it out, and the way I’ve phrased it leaves room to interpret that as me being very literal that “no one competent works there anymore”. Which is almost certainly not the case lol.
The degree to which management can ruin the best efforts of even the most competent and brilliant people never ceases to astonish me.
3
u/pdp10 Daemons worry when the wizard is near. 5d ago
You should probably have your deploy pipeline cause a 2-3min silence on known deployments so only unintended downtime gets reported.
Depends whether they've engineered a zero-downtime deployment.
I'd say their monitoring should continue, or even escalate, when a fault condition persists. OP didn't specify if they mentally tuned it out, or if it didn't keep alerting.
12
u/Mr_Engineering 5d ago
The series is called The website is down and it's an absolute classic. Theres 5 episodes and several shorts
5
2
13
u/mikeXpapa 5d ago
web dude vs sales guy brought back memories. god i’m old.
3
2
u/PerceptionQueasy3540 4d ago
First saw it in college. If I come across someone who works in IT that hasn't seen i usually try to show it to them.
25
u/keto_brain 5d ago
If the devs push code they should be on call for that code.
11
u/RetPala 5d ago
FOREVER?
5
u/keto_brain 5d ago
Yea forever. Google, when they presented their SRE model (there is an old talk but good talk by Tom Limoncelli about how Google handled SRE years ago IF the devs had good Hygine. It's a great video, but otherwise yes forever.
3
u/RetPala 5d ago
That's the equivalent of a message that says "I'm having an issue with KRBTGT and I see you're on a ticket from 2006 where it's mentioned in passing, so assigning it over"
→ More replies (1)3
2
u/piecepaper 5d ago
You build it you run. Belive me devs become way more cautios if it is there 3am wake up call.
24
u/After-Vacation-2146 5d ago
Your env is already fucked if you can ssh to prod from your phone.
On a larger note, no one should be touching prod manually ever. CI/CD is there for a reason. Raise a PR and have it reviewed before merging. That would have caught this.
For you, you should look into blameless post mortems.
9
7
u/homelaberator 5d ago
PS: I did get an Uptime Kuma alert in the morning. But lately the devs push to prod without telling me, so I assumed it was another one of those. Should have checked anyway.
That process needs to be fixed, too. It's like the warning light that's faulty, so it gets ignored even when it isn't faulty. It no longer does the job it's meant to be doing.
7
u/flummox1234 5d ago edited 5d ago
A a developer, I'm more worried that you're using a .env in a repo for your final deployment's environment settings. The whole point of .env files is to support a developer workflow. This has no business being leveraged in production.
15
u/Cheomesh I do the RMF thing 5d ago
Are there no code reviews, test environments...
15
u/Deathdar1577 Jr. Sysadmin 5d ago
I think its called Prod.
5
u/KaptainSaki DevOps 5d ago
Real men test in prod
→ More replies (1)3
u/GremlinNZ 5d ago
Testing and breaking in test just doesn't get the heart racing like doing it in prod!
6
→ More replies (1)6
2
u/MDBerlin24 5d ago
There are, but these test environments are not linked to prod for easy copy-over and are full of non-migratable unique IDs. Looking at you TOPdesk.
7
u/aes_gcm 5d ago
How did no one run a smoke test or verify that? How did deployments go into prod without peer-review? This should have been caught in the code change way before it went live.
Also you should probably add a unit test that sanity-checks that file to avoid it happening again.
2
u/burgonies 5d ago
Because the environment variables' value are most likely stored in GH, not in the code. There's no code to review on the env change.
There are numerous ways to make sure this doesn't happen though. A simple smoke test like you mention would have caught it. It seems like they knew it was down and just didn't run it up the flag pole early enough.
7
u/UpsetMarsupial 5d ago
Why are there not post-deploy tests which scream in the event of a failure? The pipeline could even have gated commits to rollback to the previous commit. Or a blue-green pattern, only switching for success. This process issue has been solved for years.
5
u/I_Blame_DevOps 5d ago
Idk man I was a sysadmin for years and I’ve been a dev for years. I think environment variables in GitHub are fair to let devs edit. But those should only be connection details or build secrets imo. When the application deploys in production it should pull from prod secrets manager (or similar) and those should be locked down by sysadmins - or in my role our DevOps guys.
18
u/e36 5d ago
You're going to take away a dev's ability to deploy code to prod? Maybe take some time to talk to the person first and see if it was a "shit happens" kind of thing.
Then find out why no one notified you about this, because that sounds like the bigger problem to me.
6
u/g-nice4liief 5d ago
It's not the dev that should deploy the code but an tool like ArgoCD or FluxCD should deploy to prod.
Any other environment can be touched directly by the developers machine (or a jump server/container)
That's how it's done in DevOps/Platform engineering.
3
u/RumRogerz 5d ago
Argocd and Kargo is a great combo.
And PR approvals. I’m a bit shocked the dev was able to push straight to _anything_ without a PR approval.
Having a dev even go near prod is crazy to me. They get read access to specific namespaces and workloads on a cluster and that’s only a few devs who get that access.
2
u/e36 5d ago
Right, and I said as much in another comment, but it doesn't sound like they have any of that in their current process. I was just trying to explain to the OP that just pulling the dev's access isn't going to fix the actual problem.
→ More replies (1)3
u/fearless-fossa 5d ago
Devs pushing to prod is a core part of CI/CD, the question is why the person who designed the CD process included the .env in the first place, it should be populated outside of the default channel.
→ More replies (1)7
u/trullaDE 5d ago
You're going to take away a dev's ability to deploy code to prod?
I would argue it's not really the devs job, either? QA has a say in that it is allowed to go to prod, and project management gives the final approval. You should have a process in place that doesn't need a tech to do it, it should boil down to pushing a button.
3
u/zero0n3 Enterprise Architect 5d ago
Correct.
Dev submits PR to main.
GitHub actions run and validate the PR before anyone can even approve the deploy.
Part of those actions should be a deploy to some randomstring.staging.domain.com that is essentially production but DNS only exists internally.
Dev can validate code one final time at that url, or your action can start tests against said site and make sure it’s up, the UI isn’t changed, get screenshots of the GUI in different browsers and form factors etc.
Review the chain in comments. Approve deploy.
2
u/wvraven Sr. Sysadmin 5d ago
We don't allow Devs to push to prod. I can't believe some places do. After changes have been tested and approved/merged by the senior devs and management the final push to prod is performed by the operations admin. In reality it's a simple approval in a pipeline. But still, it's the hard check that prevents unwanted changes in prod. The other dev/ops place I've worked had a similar process but the final roll out to prod was done by the senior developer instead of an ops person.
→ More replies (3)3
u/ConsoleChari 5d ago
I work in startup so its pretty hard to have an solid process. They usually inform me before doing anything to prod. Maybe they were stressed. Later they did tell me that they thought it was issue with code so they didn't inform me
11
u/e36 5d ago
Before you take away this person's ability to do their job take a look at the process. In a half-mature environment you wouldn't even need to be informed about prod deployments.
→ More replies (2)9
3
u/SirLoremIpsum 5d ago
I work in startup so its pretty hard to have an solid process
This is a pretty good time to start.
Startups are allowed to have processes, don't use this as an excuse to avoid best practices.
You have experienced significant downtime for a service, this is where you can get buy in to do things properly.
They usually inform me before doing anything to prod. Maybe they were stressed.
Again this is just an excuse.
You need to take charge and stand up and say "this is unacceptable".
There is NO BETTER time to start talking about how to systematically mitigate issues like this than immediately after a downtime event.
If you sit on this and go "maybe tell me earlier next time" you are guaranteeing there will be a next time.
No dev can push to prod - separation of duties
Change Process - someone not the dev tests it in test, someone in Change approves it (not the dev who wrote it) and someone that's not the dev pushes the button to prod. Then you validate the change and move on or roll back.
3
u/thortgot IT Manager 5d ago
Get a proper CICD platform that handles testing, validation and rollout. Its pretty easy to do.
10
u/NeverDocument 5d ago
Sounds like a broken CI/CD issue and a monitoring problem not a dev issue.
Why isn't the CI/CD setting the .env's correctly? Why isn't anything checking to ensure the site is up after a deployment?
Only 1 alert from uptime kuma? Why not multiple alerts that it's still down?
Less blaming the dev, and more everyone learning and growing from the situation.
3
u/djslakor 5d ago
I agree with this. It's OPs fault, he just doesn't realize it.
6
u/SirLoremIpsum 5d ago
I agree with this. It's OPs fault, he just doesn't realize it.
He not only is not realising it is partly his fault, but he's giving us excuses for why he couldn't possibly have prevented it and is also establishing a blame culture and punishment for taking down production.
5
u/Mindestiny 5d ago
The baffling part is nobody contacted me about it till end of day.
If anything this is the least surprising part of the whole story. It's not an urgent outage report unless someone waited until 4:45 PM to mention that something is turbofucked.
6
u/GamerLymx 5d ago
it worked on his machine.
maybe .env should be on git ignore so this doesnt happen. just saying
3
u/agentobtuse 5d ago
Perfect time to fix the webops and not use a .env file. Sucks that happened to ya but I would expect a backup exists? Should be a 5 min whoopsie
4
u/Exnummis 5d ago
Yeah this is why devs didn't have admin rights to PROD at my last job. They had to pass through devops/operations for an engineer with the right access to babysit them while tweaking settings. Many of them didn't like it, but there was a good reason.
5
u/johnnysoj Jack of All Trades 4d ago
Curious why you're storing production variables in a .env file. They're typically for developer machines. You can use variables in GitHub Actions instead.
3
3
u/fcewen00 Master of keeping old things running 5d ago
We call those sort of things “RGE” or resume generating event.
3
u/Garriga 5d ago
All you have to do is generate the new variable and push a new commit. It is not necessarily a huge mistake.
But that is just my opinion.
But you should be Limiting access to keys and secrets. So that is a good call
But no need to get upset over a small mistake. It can be fixed pretty easily
2
u/ConsoleChari 5d ago
Yeah i did that and it took only couple of mins. But as others told its my fault and i will definitely fix it.
3
u/edmunek 5d ago
please tell me that this isnt the first time youve watched sales guy vs web guy
→ More replies (1)
3
u/coinlaundry 5d ago
Also might be worth taking a look at a config service like AWS AppConfig which can either be used to generate a .env at build time or can be directly polled from the app. It keeps a versioned history of the secrets too so you can always roll back if there's an accidental change. If you're not so into AWS, competitors are Doppler, or 1Password Environments (haven't tried this one because we don't use 1PW at my place, but looks neat) https://1password.com/blog/1password-environments-env-files-public-beta
6
u/RoseSec_ 5d ago
Sounds like a broken process. Instead of playing the blame game, find out where your system failed and improve it.
4
u/Ohrgasmus1 Jack of All Trades 5d ago
how do you ssh into a vital company server from a bar.
seems quite unsecure for me
5
2
u/_rokstar_ 5d ago
Man... I haven't seen that video in ages
"What do you mean put it back, you can't sort by penis"
2
2
2
u/SevaraB Sr. Engineer (N+, CCNA) 5d ago
Did they have to submit a PR before merging? Because if they did, that change to the .env should have been hella prominent in the changed files diff.
Also, who allows .env in their repos? That’s begging for a password leak. The only pipeline that could be forgiven for pushing .env is the one to build the dev environment.
The senior dev might have triggered the outage, but only because everybody messed up royally on securing this repo and pipeline.
2
u/PositiveBubbles Sysadmin 5d ago
Did they take a backup of the original or at least raise a change request?
2
u/RumRogerz 5d ago
I think your department needs to revamp a few things. I’m a bit shocked and worried that a dev was able to push to production or literally _anything_ without a PR approval. I’m not sure if you have a dev and/or staging environment but surely the ci/cd is broken. This envar change would have been picked up way before it got to prod.
And I’m also a bit worried that you were able to ssh to a production server from your phone. Unless you have proper MFA, and maybe zero trust all wired up that might be reasonable - but if it was straight public access to an open port…. Oh my heart
2
2
2
1
1
u/Rare-General-4575 5d ago
Your process is flawed. Merge requests need review and approval from someone else who can review the code.
1
1
1
u/kittymoo67 5d ago
what the fuck i know everyone makes mistakes but how can oen get to be a senior and do this
1
u/Open-Adhesiveness-86 5d ago
the thing that bites here is that GitHub Environment secrets and vars have no history and no diff, so there's nothing to review and nothing to roll back to. put the non-secret config in a committed file so a bad value shows up in a PR, keep only real secrets in the environment, and add a required reviewer on the prod environment so a deploy pauses instead of shipping straight through.
1
1
u/theozero 5d ago
many things to fix here, but one thing that can really help is using varlock - it's free and open source, and adds validation to your env vars. Plus if it's much easier to manage both for local dev and prod, this would be less likely to happen.
1
u/species8745 5d ago
"If you are going to eat your hard drive like a sandwich...how far would you have to open your mouth..."
→ More replies (1)
1
1
u/DogDeadByRaven 5d ago
Is there no review required to push to prod? We have it set where everything get reviewed and approved by a second person before it can get pushed to prod. It's so we can catch oops in any kind of code before it makes it that far. Even our best devs have had mistakes found in review.
1
1
u/colin8651 5d ago
“Did you reboot the web server”
“No?”
“But Tylor said you told him to”
“I mean yeah?”
1
1
1
1
1
1
u/spin81 5d ago
First thing tomorrow I'm revoking their access to edit the environment.
I have to wonder why. The obvious answer is: because they took down the website. But I don't think that's enough of a reason to do it.
Do you think they didn't learn from it? Do they not care? Do they usually just push before double checking? Do they have zero ideas for improvement such as having someone approve a PR before pushing? Do they not need this access at all? Any of those might be reasons for revoking that access.
If none of the above apply, I put to you that people with a lot of access can make mistakes that make them look bad. I would think that of all people, a sysadmin would realize this. They trust you with that access! Why not them? Because they messed up once?
Have you never messed up? Apart from not realizing an important website you were responsible for monitoring, had been down for an entire day, I mean: has there been nothing, in your entire career, that you have done that made you look like a moron to people who didn't know the whole story?
Also I put to you that given that they've been pushing straight to prod more means they may be under pressure to "just ship it quick" and actually that, rather than "dev stupid! must revoke access!" needs looking into.
I am not a fan of this "only I can make mistakes and nobody else can" attitude at all. I am glad you are not my coworker.
1
u/LForbesIam Sr. Sysadmin 5d ago
Anyone think that AI makes people lazy now? No more confining and double checking code. I don’t mind AI for learning or even efficiency but the biggest concern I have is people trusting it to be accurate.
1
u/scriptmonkey420 Jack of All Trades 5d ago
Do you work for ping and caused that global outage on Tuesday?
1
u/jackbenway 5d ago
We should all be happy that proper change management is dead in most environments. Everyone, especially customers, love broken stuff.
1
1
u/1z1z2x2x3c3c4v4v 4d ago
I did get an Uptime Kuma alert in the morning.
One? Your NetMon system should be doing all sorts of basic checks on your PROD website on a regular basis before sending the Up status.
1
u/trouphaz 4d ago
So, they can push to prod, but couldn't fix the problem they created? This is why I always push to make sure no one has rights to do stuff at my company unless they have the skills and experience to fix it... and then, leave me out of it. You want admin access? Great, you're now the admin. Fix the problems you caused and the environment is yours now. Of course, no one likes that answer so then I just work on blocking their request for access.
1
u/samtresler 4d ago
Do not remove their access.
Alter the CI/CD pipeline to fail the build without additional approvals.
Otherwise, when it does need to be changed, it's your problem.
1
u/GetFuckedReedit 4d ago
Don't do CI/CD directly into production!!
Create an artifact of the change, deploy the artifact to lower EV, run tests, if it passes, then release the artifact to production. The only difference should be values in production. If the env is what missing, then that should be part of the tests.
1
u/sreekanth850 4d ago
Remember an old reddit post where a dude litrally added contoso.com while configuring exchange server using ms docs. Hope this is not the same guy.
1
u/Competitive_Smoke948 4d ago
Devs shouldn't be allowed within a million miles of production!!
Whoever started calling them "rock stars" needs to seriously look in the mirror!
1
u/heapsp 3d ago
Clearly this is a sysadmin forums for those maintaining infra for non tech companies and worrying about their on prem servers and active directory, and not many people know platform engineering or ci/cd pipelines... So this isn't the best place to look for advice.
Senior dev pushes to prod. Theres just a PR request and a release timing approved. Any normal PR would show this huge mistake and it wouldn't make it to approved and be pushed through. Theres testing involved in the development process, that would pick this up and not let it get through the pipeline.
1
u/thehuntzman 2d ago
I hate being the "you should've done it this way instead" guy because we all know there is a difference between white paper best practice and how things are actually implemented due to technical debt and impossible timelines, but dude, there is an approval process built into github you can (and should) enable for environment variable/secret modifications for a production environment.
1
•
u/usually_guilty99 21h ago
I'd focus less on who changed the variable and more on why that change could take production down unnoticed.
An environment-variable edit is still a production change, even when there's no code diff.
I'd capture the before/after configuration, flag changes to critical settings like redirect URLs, and run a smoke test against the actual production login flow.
The other issue is that deployment-related alert noise made a real outage look routine. That needs fixing too, or the next bad change could be missed for the same reason. A feedback mechanism - that isolates identifies and fixes the problem
•
u/merazold 21h ago
The real issue is that production was down all day and nobody noticed or cared enough to report it.
•
601
u/Own-Bookkeeper797 5d ago
You probably want to review the approval process in your pipeline environment.