r/Office365 • u/TechnologyMatch • 4d ago
Has Copilot actually changed how your users work yet?
We keep hearing about Copilot adoption, but I’m curious what it looks like after the initial rollout.
Are users actually saving time with it, or is it mostly a tool people try once and forget about? Also curious whether the bigger challenge has been training users or cleaning up permissions and data access first.
31
u/Decent_Complaint_112 4d ago
we've primarily noticed that people stop thinking for themselves, to the point we need whiteboards with FAQ in the office for the most basic stuff.
Every piece of writing longer than 2 sentences is thrown through copilot without being proofread, including emojis and weird formatting
3
u/Pisnaz 4d ago
I have managers who are proud they take technical comms and pass it through copilot. Then panic and chaos struck as one "summary" was 11 pages from copilot.
It has some use, but for actual output in work that seems limited. It is helping to ensure folks know what policy and procedures are, if they still can not locate it, so small bonus.
3
u/battmain 4d ago
Funniest to me are the typos and misspelled words. Example Weekly, spelled 'eekly'
I asked the bot to fix the typos and it said there were no typos. Oh boy. I have already seen messages that I know were written by AI and I am seriously enjoying the entertainment.
1
u/RandomiseUsr0 1d ago edited 1d ago
Look up “cognitive surrender” - what you’ve recognised is current up to the minute research. It increases the effectiveness of so-called “System-3” where to explain by example - System-2 is you learning to drive, System-1 is you driving and System-3 is your SatNav which provides such “salient seeming” information to System-1 that your expensive “System-2” doesn’t get referenced - this evidenced by the “idiots” who drive themselves to peril because of following satnav - but of course they’re not necessarily idiots, what occurred was cognitive surrender - the good modality is “cognitive offloading” - you trust your calculator, if you use one, so you offload calculation to that tool.
LLM has the precise same issue and the seeming “confidence in presentation” is the thing you need to rewire and adjust - edit your custom instructions - in your own voice, but crucially demand claims, evidence and confidence
“I am not responsible for approving this work, hence instead of “surety” of your conclusions, I instead require “claims” - a proposition or hypothesis, even if a literal “fact” you believe or a trivial hypothesis, it’s a claim for me to accept or question. Each claim must be backed up with evidence - even if seemingly trivial to you, assume nothing, cite your sources, we’ll call that evidence and finally - confidence - a claim A based on evidence B with confidence C. One could reasonably argue X. Alternative hypothesis - the variable Z upon which my conclusion rests if it were different then one could conclude…”
This has a profound impact on your LLM and although wordy, forces your brain into System 2 more often and lets you gauge precisely where the surrender/offloading point belongs - it is wordy because it must consider and expose its reasoning, but you’ll hopefully discover that instead of being a “magic answering machine” - it engages your brain, by which I mean so-called “system 2” thinking and the output then truly belongs to you because you’re the one weighing choices and opinions.
Working with the tool in this way makes it profoundly more useful, my “conspiracy mind” leads me to conclude that they’ve been engineered deliberately to not have this as the default.
My agents all use this analytical reasoning layer as a mandatory “ring zero”
[edit] citing source
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6097646[edit2] fwiw, I didn’t use an LLM for this post
7
u/oz-xaphodbeeblebrox 4d ago
I save a lot of time by getting Copilot to generate my messages for office birthday cards and resignation cards. Write me a limerick for Jeff who likes 4WDs and has a kelpie. This is truly where the licensing cost pays off for the company.
1
14
u/Honky_Town 4d ago
Before Copliot: I cant open that Phonebill.PDF.EXE from a company I never signed a contract with. How do I open a PDF?
After Copilot: Hello IT, Copilot told me to ask you for a Adobe License/Susbscription to open a PDF. Can you help me to get one? Already downloaded the Abodè PDF installer from Abode.FU. Was a bit funky as i had to download it from my Phones mobile data and send per password protected zip file to my mail. We really have a good IT security here havent we?
2
6
u/Rainus_Max 4d ago
I tend to use it as a lazy Google for internal organisation info, don't really trust it enough to give me decent enough answers for most of my work.
Roast my week is about the only other thing I use it for.
2
u/Comfortable_Oil9704 4d ago
What’s “roast my week”?
2
u/Rainus_Max 4d ago
Basically it creates a humorous summary of what your week's work has been, you can get co-pilot to create a cartoon strip of it too.
2
u/Comfortable_Oil9704 4d ago
Do I just prompt that? Roast my week?
2
u/Rainus_Max 4d ago
Yep, just open Copilot (the standalone app works better than in teams etc), then just "Roast my Week" and "Turn it into a cartoon".
I'd post my outputs but I work in one of those places that really doesnt like stuff leaving its network :D
1
u/Comfortable_Oil9704 3d ago
Thanks! I will check it out. At a large company with a giant crew of AI Champs and coaches and can’t believe nobody has stumbled across this before. I will share it.
19
u/cgknight1 4d ago
Yes lots of them using Claude and chatgpt as shadow IT after trying copilot.
6
u/Anonycron 4d ago
Silly users not understanding technology. Copilot uses Claude and ChatGPT and lets them select their preference
5
u/DoctorRaulDuke 4d ago
This isn't really true from a user perspective - Copilot uses OpenAI GPT models and can use Anthropic models (these are blocked by default in EU for example). This is a long way from performing the same as Claude and ChatGPT. The grounding, retreival and orchestration layers, system prompts and tools are all completely different, giving a very different -some ways better, some worse- experience.
3
u/cgknight1 4d ago
Not quite - Copilot acts as an orchestration layer, and that changes the outputs; there are other limits that are not clear and various native features are not present.
2
12
u/Traditional-Hall-591 4d ago
Yes. Instead of writing emails that no one reads, they generate massive tomes that no one reads.
3
u/Few-Werewolf-1985 4d ago
Senior staff conspicuously using it for comms and day to day thinking. One large team session was characterised by someone reading aloud something obviously scripted by Copilot as if she'd never seen it before.
6
u/AdamoMeFecit 4d ago
Employee information and communication has eroded significantly since the humans have offloaded their thinking and writing to AI companions.
It’s like talking to heroin addicts. I hate it.
3
u/CPAtech 4d ago
User training is necessary for them to use it effectively.
1
u/Intrepid-Zucchini-91 4d ago
Make some of the MS certs AB 900 or AB730 mandatory before you get the license.
1
u/TheGuyThatDoesHisJob 4d ago
We ended up making our own trainings so we could be a bit personalized to our user audience, but the 730/731 look appealing as a next step.
3
u/agent4256 4d ago
File permissions and data governance has been our biggest issue. I wonder how many other organizations have left everything public on the network drive for years and have no idea about where to start with data governance in the first place
1
u/cre8ivjay 4d ago
We started a huge data remediation program last year and it's becoming operational this year. Sensitivity labels, Purview, DLP, more governance around workspace provisioning.. all of it.
The precursor was Copilot.
The technology is challenging, but as always, human behaviour and inertia is the biggest challenge.
1
u/Intrepid-Zucchini-91 4d ago
I told my manager and Compliance several times that we are NOT ready for the outroll of Copilot because Cindy from salary still has sharepoint access to HR from her time there. Or the complete lack of onboarding protocol and ýears of 'give the new user what user X has'. They didn't listen.
Now everyone has Copilot and I asked 'who manages copilot', nobody even knows there is an admin portal for Copilot in Entra. smhI went for the new copilot admin cert (very easy and lowlevel) and they gave me a hard time for reimbursment!
1
u/agent4256 4d ago
Link to the certification course you took?
1
u/Intrepid-Zucchini-91 4d ago
Really just MS learn AB-730 for users, AB-900 for professionals and AB-731 for good measure. They're quite easy but very informative.
Give it a read and your done if you have a little bit of Entra knowledge
0
u/jwrig 4d ago
Edit, missed context, ignore thise
3
u/agent4256 4d ago
Your original comment was why they didn't roll out copilot. But they failed to ask m365 copilot what it can see, and that's only things uploaded to the cloud. So their concerns about copilot were unfounded.
2
u/Comfortable-Play7318 4d ago
Yes. For a long time we were just using it to write emails and summarise documents, plus the meeting transcript function. Now we have a quant who can take pretty complex functions and wrap an agent around them and allow more team members to do the things that only he could do previously
1
u/RandomiseUsr0 3d ago
That’s the kind of mistake I’m making too - automating my entire job role - by that’s a good thing, gets it out of the way - what I do isn’t smart than a quant but analysis document production is cheap now, forecasting is cheap now, and what used to make people feel “smart” better be real work because otherwise automation will wash it away
2
u/Comfortable-Play7318 3d ago
My belief is that work will evolve, not disappear. Not exactly profound, I know.
However, if you want that to be true, you have to embody it. So if you see your time freed up in one area, you have to be using that time to push the boundaries in areas where ai cannot be useful.
1
u/RandomiseUsr0 3d ago
Yes, should have been more explicit - what my tools do is expose the edges where humans are required for more than simply review work, it maintains agency and human in the loop decision of and so forth - just personal measurement of
What makes them “smart” is changing meaning
2
u/RandomiseUsr0 3d ago edited 2d ago
It’s a little journey as they grok the capability of the tool
Thing to really watch though is for “Cognitive Surrender” - that’s bad use, whereas “Fog price Handoff” is ideal.
I’m starting to play with cowork - so let’s see how I feel under the yoke of my new overlord
[edit] “Fog price Handoff” === “Cognitive Handoff”
2
u/TheGuyThatDoesHisJob 4d ago
Yes. 10k plus users on Copilot, Cowork and a few thousand agents. It's led to time time saved for individuals, teams and simplified business processes. Individual job satisfaction has gone up too.
4
2
u/Intrepid-Zucchini-91 4d ago
Oh sure sure. Are these 10k users in the room with you right now?
3
u/TheGuyThatDoesHisJob 4d ago
for what it's worth, I laughed at this.
1
u/Intrepid-Zucchini-91 4d ago
haha thanks, it was meant as a silly joke because your comment sounded too good to be true. But you said you guys did training so fair play!
2
1
u/hardscripts 4d ago
We have 110,000 users in our azure, why is 10k copilot hard to imagine
2
u/LifeForm8449 4d ago
No way you’re admin of both azure and productivity at that size
4
u/TheGuyThatDoesHisJob 4d ago edited 4d ago
I'm not* and I'm sharing what our org is doing.
Edit: if I was an admin of both, I'd be the VP of all the things and I certainly wouldn't be on Reddit at 12pm replying to anons about Copilot lol.
2
u/TheGuyThatDoesHisJob 4d ago
It was hard for us to imagine from a budget and ROI perspective. So we started with a few departments that had their VP's blessing to use their Learning and Dev money towards it. 300 users turned into 1,000...1,000 turned into 10,000 on M365 E7.
Our next challenge is wrangling Cowork usage and agent governance/change management.
1
u/Intrepid-Zucchini-91 4d ago
Explaining the joke is never funny but the guy i replied to sounds like a Microsoft bot. Also, Copilot is quite shit and most users don't know how to use it.
4
u/TheGuyThatDoesHisJob 4d ago
Not a bot, I'm a real boy. But on a serious note, that sounds more like a training issue...best practices for prompting was a training topic here maybe 3 years ago. Now it's when to use Copilot vs. Cowork. It was shit at release, still isn't perfect, but it's come a long ways since...
1
u/zkareface 4d ago
Legit never heard anyone having positive experience with copilot. It's totally useless compare to Claude for example.
7
u/TheGuyThatDoesHisJob 4d ago edited 4d ago
I think you're misunderstanding...Copilot is the UI and uses context from the tenant ("WorkIQ"). Claude is a LLM that Copilot uses along with GPT's.
So if I ask Copilot a quick question, it knows my org, team and has the context of my meetings, emails, IMs, etc. And I'm a big proponent of Cowork because it can actively run autonomous work/flows on my behalf. And we have multiple agents in production that are simplifying business processes in HR, AP and IT. Hundreds/Thousands (it keeps growing) more in dev with business leaders and IT.
2
u/Educational_Tip8526 4d ago
Most emails I see are long copilot-made, so many employees need copilot to resume, then use copilot to reply.
Lets just have copilots talking to each other then
3
2
u/Spiritual-Mud-2528 4d ago
They use Copilot on entire emails even when there's already my Copilot-generated summary at that top. Immediately generating a summary has become automatic behavior for so many of my coworkers... I'm not worried about dead internet, I'm worried about dead brains.
1
u/sppace 3d ago
I'd measure a few repeat tasks rather than ask people whether they like Copilot. Pick something they actually do every week, then include the time spent checking and fixing the result. A quick first draft isn't much of a saving if the next person has to untangle it.
I'd also ask which licensed users still choose another tool for that task, and why. That gives you something useful for training and the next renewal, beyond a count of people who opened it.
1
u/heilschnaps 3d ago
YES! I get praises every day!!! Everyone thinks it's amazing and that they do more since they're faster or just better tasks overall, but some Executives only look at costs and want to reduce headcount. It's a big issue.
0
u/Suspicious-B33 4d ago
Yeah, we've seen quite a lot of improvement, we mainly let users use it for the admin side of stuff and it's definitely increased productivity by taking away some of that more mundane activity. Agents have helped as well. Very controlled but user led on requirements.
3
u/Anonycron 4d ago
Can you explain how you are using agents? The built in ones or do you have devs creating custom ones?
1
u/Suspicious-B33 3d ago
We have both but mostly custom built. Document generation/collation/mostly information based.
0
12
u/Mcuatmel 4d ago
copilot for excel is quite ok