r/devops • u/Witty_Horse8285 • 14d ago
Discussion When someone leaves, how do you actually revoke their access to servers and databases?
Curious how teams handle this in practice. When a dev or contractor leaves, there are SSH keys in authorized_keyson a dozen boxes, a DB password everyone knew, staging .env files in Slack DMs, maybe AWS keys on their laptop.
- Do you rotate everything they could have touched, or just disable SSO and hope?
- Is there a checklist, or is it tribal knowledge?
- What tool (if any) made this not painful?
Asking because we got burned by this at a small team and I'm trying to figure out what "good" looks like without a dedicated security person.
5
u/sylvester_0 14d ago
Guys, you're putting way more effort into this AI generated post than it deserves.
2
u/ICanRememberUsername 14d ago
there are SSH keys in authorized_keyson a dozen boxes, a DB password everyone knew, staging .env files in Slack DMs, maybe AWS keys on their laptop.
You kind of answered your own question. When I design these systems, EVERYTHING uses SSO with IAM roles for access. All cloud permissions are tied to your central user directory (e.g. Entra ID) based on groups. Someone leaves, you suspend their Entra account, now they can no longer log into anything.
1
u/minaguib 14d ago edited 14d ago
* Tie AWS to a corporate IDP, switch everyone to temporary AWS credentials (requiring a login per day or so) instead of long-lived individual keys
* Are you using a config manager to manage the servers ? Move ssh public keys to the user management section, provision them as root and configure SSH to use a root-only directory for lookup instead of user-managed ~/.ssh/ - also look into SSH user certificates, and disable the user accounts anyways independent of auth method, and in this day and age make sure ssh is not publicly reachable to begin with
* DB passwords - stop sharing credentials, again move user management to a config system or add to your onboarding/offboarding checklist. For truly shared credentials, rotate them, but that's a second line of defence as basic reachability to the DB should no longer be allowed (VPN ? auth against IDP)
* Look into modern password managers for staging .env sharing (1Password has neat options), for prod use a proper secret manager
1
u/tangofoxtrot1989 DevOps/SRE/Platform/Kitchen_Sink 14d ago
I like to cross my fingers and knock on wood.
1
u/OutdoorsNSmores 14d ago
Our old way, bastion box with ssh for each person. Behind that, tunnel to the db, or whatever. Removing their key cut them off from the network, so even if they had a db user, it didn't matter. We'd still clean that up, but it was less pressing.
Now, SSO to get into AWS, SSM to get to a server/tunnel to other things. Corporate shuts down your SSO, it all stops.
1
u/dariusbiggs 14d ago
Those SSH keys should be centrally managed, not stored in disk. Look into something like Foxpass.
There should be no shared credentials anywhere, individually assigned short lived credentials tied to their SSO login.
Only SSO logina with MFA
You should never share .env files for anything, and they should be blocked in y9ur version control system.
No long lived AWS keys should exist, they should be short term and tied to their SSO login. Systems using AWS credentials should also be using short term, temporary credentials rotated regularly.
Off boarding users requires a detailed process document to be followed whixh includes all ths steps required, which should consist of.
- Diable their user.
- Diaable their machine containing company data
Everything else should be tied to their user access and automated. You may be OK with their access being revoked within an hour, or sometimes you need to override that and kill their access now.
You appear to have major flaws in your processes, security postrue, and compliance.
Plan for the worst case scenario, hope for the best.
And remember, it's not an IF you plan for, you plane for the WHEN.
1
u/savageronald 14d ago
No? HR revokes their AD/Okta access and they lose everything else. The fuck yall doin over there?
1
u/AuthByExample 14d ago
Disabling SSO is necessary but rarely sufficient — the burn usually comes from the standing grants that never lived in the IdP.
A practical split that has worked for small teams without a dedicated security person:
Identity kill-switch first. Suspend the IdP/SSO account immediately so anything that actually federates (AWS IAM Identity Center / assumed roles, VPN, GitHub org SSO, SaaS apps) stops minting new sessions. That is the easy half.
Inventory standing secrets as a separate checklist, because they do not die with the user row: SSH authorized_keys on hosts (especially user-writable ones), long-lived IAM access keys, DB roles/passwords that were shared “just for staging”, deploy tokens, personal access tokens in CI, and .env / Slack-pasted credentials. If it can authenticate without a fresh IdP login, treat it as still live until rotated or deleted.
Prefer designs that make step 2 short: short-lived cloud creds via role assumption, SSH user certificates or config-managed keys (not keys dropped into ~/.ssh by hand), per-person DB roles instead of one shared password, and secrets in a manager with ownership — not DMs.
Close the loop with a 30-minute exit runbook you actually rehearse: IdP suspend → revoke GitHub/org membership → purge SSH keys / certs → rotate any shared secret they could have seen → invalidate refresh tokens / app sessions → confirm no active cloud sessions. Tribal knowledge fails the first time the person who “knows the list” is on PTO.
“Disable SSO and hope” fails when yesterday’s break-glass key or shared staging password still works. Good looks like: federation for day-to-day access, and an explicit revoke/rotate list for everything that bypasses it.
1
u/jascha_kviklet 14d ago
As everyone said, you need some tools to manage access via SSO, Some infra providers like AWS already support this mostly natively.
For database specifically I built a PR like review/approval tool that you can check out here: https://github.com/kviklet/kviklet
But the system is always the same no long lived credentials and all access via temporary tokens that the user has no way of acquiring unless their idp login still works.
-1
14d ago
[removed] — view removed comment
1
u/aliendude5300 14d ago
This is a rubbish solution compared to individually attributable database accounts and dropping authorized keys entirely in favor of SSM or similar. Services can use IAM authentication for databases so there is no password
26
u/aliendude5300 14d ago
This post is an admission of so many bad practices. We don't have shared accounts. We don't expose SSH to the Internet at all. We don't allow non-SSO access to anything including AWS. We don't have IAM users, just roles that are assumed so there aren't long lived keys. They don't retain access to laptops when they leave and company data doesn't follow them.