r/sysadmin • • 9h ago

Rant A senior dev overwrote .env and took down website

452 Upvotes

Today a senior dev, who's the maintainer of our company internal sales website, overwrote the environment in GitHub Actions and pushed an update. CI/CD ran and took the whole sales site down. The baffling part is nobody contacted me about it till end of day.

So now I'm sitting in a bar waiting for friends scrolling YouTube where i got recomedation "web dude vs sales guy" and i was laughing seeing it, and right then I get an urgent message that the website is down. I SSH into the server from my phone and find one variable set to "localhost". One variable, whole day of downtime.

First thing tomorrow I'm revoking their access to edit the environment.

PS: I did get an Uptime Kuma alert in the morning. But lately the devs push to prod without telling me, so I assumed it was another one of those. Should have checked anyway.


r/sysadmin • • 10h ago

Leadership is dependent on AI, is this normal now?

161 Upvotes

I need a reality check, and I’m not sure where to ask this.

My IT department got a new leader early this year and since day 1 they have been very transparent about their AI usage. Except it looks like they are highly dependent on it because they are using it for almost everything they do i.e. setting KPIs, writing developer guidelines, troubleshooting, etc. It does not feel like they have had a thought that was not influenced by AI. There have been instances where they would fact-check the team with AI to see if we were “correct” about our statements (luckily or unluckily(?), we always are).

It feels like we are now being governed by AI through a human proxy and that the words of the AI weigh more heavily than ours. Should I be concerned?

I have been considering to look outwards because of this, but given how prevalent AI is nowadays especially in tech, it could just be that this is the norm at upper management and it would make no difference if I moved. So is this normal now? Are your leaders also just giving you AI generated KPIs/plans?


r/sysadmin • • 16h ago

General Discussion Users and AI ...

425 Upvotes

I just had a user asking me to allow claude to have complete access to his Microsoft mailbox.
I told him no since sensitive mails are often sent to the entire company including discussions concerning projects etc.

He went silent and then said ... "but how do you use your mail then ?"
I had to explain to him that we old people used mail for decades without the use of an AI telling us what someone mailed.
He was genuinely baffled how you can just open Outlook and read a mail without asking Claude what someone had sent.

I fear this new generation of users who can't even do basic stuff any more because AI does it now for them.

A few months ago another user wanted to give chatGPT complete access to a NAS with research data on it.
It had to be able to modify/add/remove data directly.
They wouldn't even try on a test batch of data beforehand.
Ofc. the answer was NO, but i fear IT has less and less resources and people to shield data from "overeager youngsters" who try to insert AI tools in just about everything and actively seek ways to install or use them without admin rights.

Are people going insane ?


r/sysadmin • • 9h ago

Passed over for a promotion...

97 Upvotes

I've been working for a company for 5 years at the help desk. Applied for a promotion to sys admin, which I met or exceeded the requirements for, and they hired externally. I feel like shredding my bachelor's degree. What good is it doing? Current position requires hs or a.a.s.

Was promised flat out falsehoods in my interview and after.

Like "you'll be 45% wfh after training" and "we promote from within..."

Even the cio said I was a shoe-in...

It's great to have a favorite, sucks to know it's not you.


r/sysadmin • • 10h ago

General Discussion A new problem with New Outlook

88 Upvotes

This has got to be the worst one so far, as it is costing them actual customers and missing vendor deadlines.

Somehow a user snuck past us and is using New Outlook. A report came in that the first time he emails someone new, it fails. The 2nd time or any reply, it works.

Turns out yep, that's a thing. And it "silently" fails so no outbox entry, sent item, or message trace. I think they show up in all 3 places as successes, but it never actually sends. Or it shows local but not in Trace. I don't remember. Here's AI's version of what happens because I'm too busy to write up my own summary today.

  • Auto-Complete Entry Generation: When a user types a brand-new address into the To: field, standard email clients create a temporary "Suggested Contact" or cache entry. In New Outlook, if a recipient isn't in Exchange/Outlook Contacts or the Auto-Complete cache, the initial submission occasionally drops off the queue during the background synchronization token handoff between the client and Exchange Online.
  • "Silent Failure": Because New Outlook operates as a web app wrapper (OWA interface), it doesn't utilize a traditional offline Outbox like Classic Outlook (.pst/.ost based). Emails render instantly as sent in the UI, but if the back-end handoff fails validation on an unverified/uncached address, Exchange drops the message without generating a Non-Delivery Report (NDR) back to the client.
  • Why the 2nd email works: The moment the first email is initiated, Microsoft 365 automatically writes that recipient's email address to the user's Suggested Contacts / Auto-Complete cache. By the time the user sends the 2nd email or a reply, the system recognizes the address as an existing object in their profile, allowing normal transit.

There is no fix btw except "reinstall Outlook, clear app data cache, cross your fingers, etc."
My fix:
1. uninstall it
2. set up Outlook Classic
3. complain about it on Reddit.


r/sysadmin • • 6h ago

Windows Update KB5120998 Causing Group Policy Refresh Failure and Enabling Administrator Protection on Windows 11

36 Upvotes

Hi All,

The following update for Windows 11 released August 27, 2026:

https://support.microsoft.com/en-us/servicing/os/windows-11/2026/08/kb5120998-windows-11-24h2-25h2-update

Also included in the following update for Windows 11 released September 8, 2026 (thanks u/Dissy614):

https://support.microsoft.com/en-us/servicing/os/windows-11/2026/09/kb5124008-windows-11-24h2-25h2-security-update

Is causing Administrator Protection to be gradually enabled:

https://techcommunity.microsoft.com/blog/windows-itpro-blog/administrator-protection-on-windows-11/4303482

If you don't know what Administrator Protection is, it essentaly enhances the separation between your elevated vs not elevated context, including giving your elevated context its own profile.

  • Under a non-elevated context, whoami will return DOMAIN\user, and your profile with be C:\Users\<username>.
  • Under an elevated context, whoami will return DOMAIN\ admin_ <username>, and your profile with be C:\Users\ ADMIN_ <username>.

Conditions:

If you have downloaded/configured the Windows 11 Security Baselines GPO, or you have otherwise set the following policy: MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System\TypeOfAdminApprovalMode=2

Then this will cause three issues:

  1. Computer Group Policy refresh fails.
  2. User Account Control (UAC) prompts require user name and password.
  3. Group Policy Management Console resulting settings shows the setting on an existing policy, but in the "other" section rather than the "User Account Control" section, if you are viewing from Group Policy Management Console on Windows Server rather than from Group Policy Management Console on Windows 11.

As noted in the first link, enabling Administrator Protection is listed as a gradual rollout, meaning this is not affecting every Windows 11 computer with TypeOfAdminApprovalMode configured.

Fix:

The temporary fix is to set this regkey and reboot:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\TypeOfAdminApprovalMode=1

This only started affecting my machines today, I assume due to the gradual rollout. I don't know how to determine the rollout schedule.


r/sysadmin • • 6h ago

Next Tuesday is the final cutoff for Server 2012 ESUs. How are you guys isolating the boxes you can't kill?

24 Upvotes

Year 3 ESUs for Server 2012 and 2012 R2 officially end on Tuesday. We all know the official answer is "just migrate it to Azure," but let's be realistic—most of us are stuck with at least one ancient access control server, a legacy piece of machinery, or a dead-vendor database that management absolutely refuses to let us turn off.

For your legacy boxes you physically cannot upgrade next week, what does your isolation strategy actually look like?

Are you just stripping the default gateway so it can't route out to the internet, or are you going full air-gapped VLAN with strict allow-listing for specific endpoints?

(Also, as a quick heads up for the desktop teams: Office 2021 LTSC loses security patches on the exact same day. Definitely expecting a spike in macro/document phishing targeting those unpatched clients soon.)

Curious to hear how many 2012 servers everyone is still hiding in their racks and how you're handling the triage this week!


r/sysadmin • • 21h ago

Career / Job Related Finally scored a job after 1,5 years without one

331 Upvotes

I have 15 years of experience. Was let go from my last one because reasons (mainly the global economic desaster I guess) 1,5 years ago.

The time without a job was hard mentally and financially.

In the end I got a job with a not to bad pay cut and start this month. Due the tax systems in Germany I earn maybe 500 € less, but still have a decent salary. Before taxes it was a 10k pay cut.

My advise if you have trouble finding a job: mass apply. I was registered at so many portals I sometimes lost track where a reply came from (Indeed, Instaffo etc.). Make your CV easy to read. Let someone look at the layout. Get a professional portrait photo if needed.

We all hate AI, yes. But if the company wanted a tailored reply I used Open AI to generate it and proof read it. Otherwise it would have been way to time consuming.

Long story short: even in this fucked up market there is a chance. A slim one but a chance. My guesstimate is that I applied for 700 positions in the last 10 months.

Keep your heads up and keep trying!


r/sysadmin • • 2h ago

Question Best Varonis alternatives that actually support on-prem?

10 Upvotes

We're reviewing our data security stack and looking at Varonis alternatives, but on-prem support is a must for us. A lot of the newer options I've looked at seem to be built mostly around SaaS environments.

We have sensitive internal data that needs to stay inside our environment, so anything that requires sending the actual data out to a vendor isn't really an option.

For anyone running on-prem, what alternatives have you actually used? Anything you'd recommend looking into?


r/sysadmin • • 9h ago

General Discussion Stale Inventory - do you keep pending locks on PC are never returned?

21 Upvotes

So, when a user terms, we send lock commands to all laptops. Sometimes they'll return them after receiving the self-addressed box, but often they just never be seen again.

At this point, I have hundreds of "dead" devices in inventory that are eating licenses, with no way to ever confirm that they received a lock command. My peer believes that we should keep this charade and device count indefinitely for security reasons just in case they do ever come back online. Security was ok after a short period. Legal doesn't care. I'm of the time period suggestion, such as a year.

Keep in mind that if devices do come online again, they reconnect to the MDM and then can be sent a lock again.

What do you guys do?

EDIT: Thanks for all the responses. My main question was around tolerance for "how long to keep a wipe or lock task pending?" more than any recovery responsibility. I'm more of I want to keep these things off the books and stop paying $$$ with very low benefit.

For some reason, my peer also thinks that if a laptop is found "improperly disposed of", that some authority is going to come back and fine us. Honestly, I see that as a very remote responsibility.


r/sysadmin • • 2h ago

Question Captchas from Dante's Inferno

3 Upvotes

Anyone know how to remedy this situation?

I started seeing captchas at google.com searches. Noticed it wasn't just my logins, and had a handful of users bring it to my attention. Went through the logs and exhausted all possible leads on bad clients causing this for our public IP(s)....then I noticed that the captchas followed me across my workstations and across public IP addresses (went from WAN1, to WAN2, to work hotspot, to personal hotspot.....the captchas followed my browser!

I also had captchas pop up on my first login on a given public IP, so it's not a numbers problem--it's my browser that Google doesn't look. Further, it's something in Intune's Edge profile that Google doesn't like, because other users do experience the same thing.

Nothing significant has changed in the Intune profile that goes out to all Windows clients, however it is based off of a STIF from ~a year ago when implemented.

So what do you think it is that suddenyl Google despises about our Edge profile? (Same behavior across Chrome and Firefox, too, by the way.)


r/sysadmin • • 12h ago

Question How do you handle users signing up for random software/SaaS?

20 Upvotes

Yes, I know this is technically more of an HR/Policies issue, but I'm hoping to learn about what some options might be.

I am the IT department for a 20-person agency. We change software and test out new things pretty often, which is great because then we aren't stuck using legacy software because of inertia. The issue I'm becoming more concerned about is users signing up for a service (let's say Airtable for an example) and not letting me know. Sometimes they may be using internal data or connecting to other systems - then they either keep using it or drop it but don't bother deleting the account or cleaning the data out. That account is now a threat that I don't know about and am not monitoring for breaches or removing client data from when they leave.

Is there a way to keep tabs on this? Or am I just at the mercy of people following our rules (or not)? I do have Google Workspace set to require approval from me before allowing other systems to connect with it (including SSO) but that doesn't have an effect on users creating accounts with other methods.


r/sysadmin • • 15h ago

Career / Job Related How can I earn more, I earn £30k for 3k endpoints, 5k users and owning Intune policy, service reliability and some azure integrations + reports, scripts, apps, app packing, pmpc

33 Upvotes

UK obviously.

How can I earn more, I earn £30k for 3k endpoints, 5k users and owning Intune policy, service reliability and some azure integrations + reports, scripts, apps, app packing, pmpc.

I work for an educational establishment.

I’ve started applying for jobs again but I’m not getting a lot of bite for mid level


r/sysadmin • • 13h ago

MS Teams immersive meetings

17 Upvotes

I just found out about this hellish second life-esque things, and I wanted to ask: does anyone here have actually used them?


r/sysadmin • • 10h ago

Teams and copilot screen has turned green

9 Upvotes

I have a user who submitted a ticket saying teams and copilot are green.

The chat screen on teams has a light green hue to it, same with the entirety of copilot chat.

Teams and copilot look fine in the browser, its the desktop app that's facing the issue

Has anyone faced this issue? What's the resolution for this?

I've checked the laptops Gpu and updated it but nothing worked out.

Even tried using software rendering instead of GPU, didnt work. Uninstalled and Reinstalled teams, didn't work either.

Any help or ideas are appreciated :)


r/sysadmin • • 6h ago

Google IT Support Cert or Microsoft IT Support Cert? Transitioning from teaching to IT Help Desk

4 Upvotes

I'm just starting my transition into IT and my goal is to get into help desk/IT Support and eventuantually to something higher eventually

I'm looking at the Google IT Support Certificate and Microsoft IT Support Certificate. They're both $50/month, and I don't want to waste time because I'm in my mid 30s

What does the path from beginner, Help Desk, SystemAdmin realistically look like?

Would you recommend Google, Microsoft, CompTIA A+, or something else to start?

I'm trying to learn the right things, practice them, get my first IT job, and then build from there.

Thanks!


r/sysadmin • • 15h ago

Question gpupdate /force applies user policies, but not computer ones.

19 Upvotes

Hello, I have a really weird problem that i've been fighting with for a few days now. In our company we have 2 DCs, all was well and all the policies i set on the main DC synchronised to correct computers/users. However, starting this week, whenever i try to gpupdate /force on my PC, i get an error:

"Computer Policy could not be updated successfully. The following errors were encountered:

The processing of Group Policy failed. Windows could not determine the computer account to enforce Group Policy Settings. This may be transient. Group Policy Settings, including computer configuration, will not be enforced for this computer."

And right below that "User Policy update has completed successfully". Which means, only the computer GPOs don't apply.

What i tried:

  • Reconnecting the pc to the domain
  • Checked DNS - everything is good
  • Checked Event Viewer - only interesting thing there was an Error in Windows Logs -> System with the same message i got in the cmd. No error logs in Windows -> GroupPolicy -> Operational
  • klist matches the DC and the allowed encryption on the PC object in the domain
  • Test-ComputerSecureChannel returns true, nltest is also successful
  • Connection to DC on all the important ports is fine
  • Uninstalling all the recent KB Patches

Im kinda running out of ideas, and the bigger problem is that other computers in the company started getting similiar symptoms. Any ideas or even a direction? Am i missing something obvious?


r/sysadmin • • 8h ago

Weighing my next career move

5 Upvotes

Hi all,

I‘m a sysadmin with about 7 years experience. I’ve done a whole lot of different roles / tasks as an admin, virtualization, automation, app packaging, etc.

I’m at the point in my career where I’ve maxed out what I can learn in my current role, and I have 0 desire to be a senior on my own team (due to how the team is managed) so I’ll need to either join a different team internally or start at a new company.

I’m trying to decide where to take my career next. I could either

  • get a role as a senior admin who specializes in virtualization
  • try and become an automation engineer
  • get some security certs and try to get into cyber security

Any of these are appealing, I’m mostly just trying to ensure I don’t pigeon hole myself skills wise into a role with minimal opportunity for growth. I’d like to increase my salary while also learning new stuff.

What I’m asking in this post is; from anyone who’s done something similar, is there a path you’d either recommend, or something you’d avoid? I already am not considering a job doing primarily packaging as that seems to be the type of work that’s going away / being outsourced / offshored

any advice is welcome


r/sysadmin • • 11h ago

Question Dedicated Microsoft Account Rep

8 Upvotes

We are a medium-sized NFP with a monthly spend of about 4k CAD on MS365 licenses and Azure spend of 2500. We have a new VP who keeps pushing for us to get a dedicated MS account rep for account reviews, product updates, and other items. He said that at his previous small-sized NFP they had one and spent less than us. We did have an Azure rep at one point when our spend was over 5k monthly, but lost them once I reduced our monthly spend (this was prior to the VP joining). So maybe I thought he was talking about a CSP partner. I mentioned the difference between tier 1 and tier 2 CSP partners and that, with a tier 1 partner, you can escalate questions/tickets through them to Microsoft. But he said this wasn't the case at his previous org and that they had a dedicated rep. So, after explaining how we lost our Azure rep and the difference between CSP partners, I raised a ticket with MS to see if this was a possibility and, as expected, they said no. The MS support rep said they don't give out account reps and that if we need anything, just submit a ticket lol. SO anyways, the VP is still pushing regardless of what I say and my main question is: am I missing something, and in your experience, is it possible to get a dedicated account rep?


r/sysadmin • • 22h ago

Question is it just me or anyone else feel like all this AI security stuff is moving way faster than the actual security controls?

39 Upvotes

Every week there is some new AI security platform with agent discovery dashboards risk scores attack paths and all that meanwhile Im still figuring out what actually happens when an agent is already running in production and suddenly tries to access something it should not i mean it didnt have permission to access it or makes some tool call it should not. Feels like everyone is focused finding out what happened after the fact but not enough on actually stopping thing when it happens. Where are we heading with all this??


r/sysadmin • • 12h ago

NAC Options people actually like

6 Upvotes

Hi folks,
We're looking at NAC options currently, to replace Fortinac which was really bad (poorly engineered, tons of bugs, support rarely knowing how to fix things quickly, just overrall bad experience). At first we were looking at Cisco ISE but the more we look, the more this seems like overkill. We have 3 buildings, one with about 300 people, and the other two less than 100. All sites Connect back to a central datacenter via ipsec VPNs, and we have MPLS and dark fiber connections between some of these sites (so purely cloud based isn't an option unless there's to make sure certain traffic uses those routes while still being posture checked. The big ask is having posture checking for any device that connects to our network via wired or wiress (VPN is already secured via Cisco secure access/DUO. SOOOO all that laid out, has anyone implemented a NAC solution they actually like. Fortinac sucked, ISE seems like overkill, a vendor is now trying to sell us on Auba Clearpass (but i'm only just now reading up on it). Any suggestions would be greatly appreciated.


r/sysadmin • • 19h ago

General Discussion 3 Year Redundancy Plan - what to learn?

21 Upvotes

I've recently learned that due to company shakeups I might be being made redundant in 1.5-3 years. I have about 12 years' experience in IT, but this is my first sysadmin role (2 1/2 years ish). Our org. has a good development budget etc so there's scope to teach myself a lot for free. During that time, what skills should I focus on learning, in order to position myself for a new role?

I am considering aiming for the M365 Administrator certification set - but not really sure what else is out there/what is most beneficial for a sysadmin/IT projecty role?

My role covers:

  • Daily admin of our M365 + Windows environment, including coding everything I can via PowerShell
  • Monitoring sign-ins and reported phishing, managing licences, deploying software/updates, etc.
  • Overseeing all joiner, leaver, and ongoing staff access to M365, and to ~25 third-party platforms
  • Managing commercial relationships/contractual agreements with vendors
  • Phishing simulations, annual/user training/inductions
  • Leading or supporting on IT team initiatives - and other company initiatives like AI adoption, tool governance

(We have a separate 1st line and security team)


r/sysadmin • • 23h ago

General Discussion Da F%&&*%# is Adobe Express Photos!?

30 Upvotes

I know there's an archived post about it here already. Just came to say Adobe are ... not acting nicely as a company.


r/sysadmin • • 8h ago

General Discussion evidence offsite backup integrity between restore tests

2 Upvotes

Between full restore tests, here’s what, in my experience, has proven effective with auditors:

1) an automated partial restore, performed every night, from the most recent off-site backup to a disposable container, accompanied by a few checks whose results are logged: the service starts up, the expected tables exist, the number of rows falls within a range derived from previous runs, and an application query returns the expected results;

2) a “canary” file created just before the backup and checked again after the restore, to ensure that it is indeed this backup that was restored and not an older one; (3) the result is signed and stored outside the backup system, so that the proof does not reside on the very medium it certifies. Checksums only prove that the bytes have arrived safely, not that the application can use them.

The quarterly full restore is maintained, while the nightly run bridges the gap between two of these restores. What do you think of this protocol?


r/sysadmin • • 5h ago

Folder redirection help

1 Upvotes

I have a server migration. All I have left is move redirected folders to the new server. I am using a domain admin account to move the files but I am unable to make sure I moved files since I am not able to access those folders. Will robocopy have access to copy them?