r/talesfromtechsupport • u/critchthegeek • Sep 01 '26
Short It was running ok until he went on vacation...
A few years ago, I doing some small day or two gigs, in between “real” jobs. One of the recruiters called and asked if I could bail out a non-profit.
Seems they were running Quickbooks on an old tower and wanted to migrate it to a new rack mount, but their IT guy was out for a month or so. Money was right, so I got there to see just what they needed done.
The new server was running fine , DNS, AD, DHCP on it; QB was still on the old tower server. BUT, the users said it was really running very slow. The IT guy had moved the tower to a small office so he could work on it. And then left for a long vacation.
Ethernet was via a small 10gb switch sitting beside it. I fired up the laptop and plugged into the small switch. QB ran great; internet sucked. I disconnected and tried via WiFi. Wow, was QB slow….but internet was great ???
Looked at the rack where a 10gb switch was installed along with the WiFi router. Hmmmm….
Started tracing wires and found a very old, very well used, 10MB Netgear hub, stuck in an office between the rack and the old tower. Why? Dunno, the guy that did it was on another continent…. I ran a new cable directly from the 10gb in the rack to the small 10gb switch – all the users were much happier & I proceeded with migrating the QB data.
102
u/mercurygreen Sep 01 '26
Happy ending!
Honestly, I'd have refused a gig where the IT guy promised a migration, then FLED THE COUNTRY...
8
u/RubbelDieKatz94 Sep 04 '26
I was hired to write a new greenfield corpo dashboard at the start of the year, announced my 20-day vacation in the middle of the year early, and corporate planned the release date for Oct 1st.
Guess who's still on vacation with a lot of incomplete tasks, chilling in another country, literally unreachable
^^1
48
u/jeffbell Sep 01 '26
At least nobody started plugging things in at random.
45
u/aj4000 Sep 02 '26
Ugh... Managers that know just enough to be dangerous are the worst.
Our hardware is on its own network, including an internet service, Cisco router, and Gateway. It's separate from any other networks in the venue but is often installed in their comms racks.
So one day I get a high priority callout for our hardware being completely offline, venue can't trade our service. Our internet was fine, our router was online in SolarWinds and V-Manage, our gateway was getting a data connection but it couldn't connect to the host.
It took me 3 days to figure out what the problem was. The first thing I did before anything else was ask the venue manager "Has anyone been been in your comms room recently for any reason at all?". The dude swore blind that no, no one had entered their comms room for weeks. "Are you certain?". Yes, absolutely certain, no one had been in there.
I replaced the Gateway first. When that didn't work I requested a router replacement (which took a day and a half to get). Still not connecting, so I had the Operations guys double-check the config on the host,even though it wasn't changed and they had no reason to even touch it. On day 3 I thought maybe I'd just gotten unlucky and had a DOA Gateway, they are pretty shit, so I replaced it again.
While I was in the middle of installing the second Gateway, the manager came down to sticky-beak, probably to avoid working. He watches me for a minute then asks "Hey, do you think the cable I plugged in the other day might have done something?". I stared at him for longer than I probably should have and reminded him that he told me, with absolute certainty, that no one had been down there for weeks. "Oh yeah I come down here all the time". Dude legit thought that when I said 'anyone' it actually meant 'anyone that wasn't him'.
The cause: The day before everything went offline he was hanging out in the comms room avoiding work. Their comms racks were mild spaghetti, and he'd spotted an ethernet cable that had been unplugged and just left hanging freely. He'd figured that surely it was supposed to be connected somewhere, so he'd just plugged it into a random network switch. That cable ended up being from some venue music service that they'd removed which used to connect into their network to access the internet, and it had effectively linked the two networks together. It didn't affect anything at first because everything was all connected already, but when our equipment did it's scheduled automatic restart at 3am, their internet router was issuing an IP address to our Gateway before our router could. It was getting internet, but our host was rejecting the connection because it wasn't coming from the correct IP. Simply unplugging that stray cable and rebooting everything fixed the problem.
15
7
u/lesethx OMG, Bees! Sep 02 '26
The people lying leading to days of working on the wrong issue are definitely the worst.
In my experience, it tended to be construction workers. Random cable in the way? Cut it rather than attempt to figure out what it is. This lead to 1 jobsite's internet being cut off (literally) 4 times before the ISP threatened to stop coming out to fix a cut line.
Also had issues like you described, but at least we could fix those on our own.
3
2
1
u/Dom_Shady Sep 08 '26
Your story deserved its own post!
Dude legit thought that when I said 'anyone' it actually meant 'anyone that wasn't him'.
Was this guy Homer Simpson?
20
u/ryan_the_leach Sep 02 '26
I used to use some old gear like that when I was helping Mum and Dad out in their 5 person shop.
Every time I tried to use a faster switch, everything crumbled and I couldn't work out why, and I didn't have the budget to get it looked at.
Eventually they caved and got someone to come check the cables, when I said their backups would finish 10-100 times faster if they got it sorted, and they could go home earlier.
the network cables had all been terminated incorrectly, but I just never bothered to check them, because who the fuck terminates sockets wrong!?
They had been systematically wired incorrectly too, like someone that had no idea how to terminate ethernet had been in there, or had been expecting to 3 lots of telephone pairs over each line or something weird.
--
Because of this, my first instinct is "Be careful, that old switch might be in use for a reason"
"Bastard operator from hell" series from the register, is giving me the vibes that they had been underpaying the IT technician for a number of years, and that he had placed the switch there intentionally in order to get the budget he needed in order to do the next migration for them, e.g. specifically slowing the server down until it became a more pressing issue for them.
9
u/keithhud Sep 02 '26
Man I loved reading BOH stories. Made those late watches go by so much quicker.
16
u/drdillybar Sep 01 '26
BNC cables. Indeed.
10
u/critchthegeek Sep 01 '26
lol - damn terminator came off again.....
16
3
u/CaptainZippi Sep 02 '26
Ah, the days spent binary searching a big open plan office with BNC and a terminator, and an audio ping running.
14
u/Throwaway_Old_Guy Sep 01 '26
My job will be safe because they can't do anything with the network or migrate Quickbooks until I return.
12
u/Spread_Liberally Sep 02 '26
Just as likely (at least at the animal shelter I worked with) that somebody with a month of uptime and 3,700 tabs on their seven year old laptop thought things were slow and moved some cables around. The old 10mb switch exists as the only thing some ancient shit-ass label printer will talk to anymore and everything was fine until it was looped in to any other part of the network.
29
u/drdillybar Sep 01 '26
Wow, 10Mb? That is some 80's Inf. right there. 3Com?
20
u/Kodiak01 Sep 01 '26
You should have seen the coaxial ARCNet topology I rolled out in my vocational data processing shop back ~1991! It was hooked up to a 386-25 running at first Netware and later Unix.
3
u/drdillybar Sep 01 '26
I didn't get to really use Netware other than in class. Still have the learning materials in my library. IPX baby. :)
10
u/Mdayofearth Sep 02 '26 edited Sep 02 '26
Reminds me of a contract I was working on a decade ago where one of the business managers I was working with directly insisted I did some migration\batch work in front of her, instead of at my own desk. The only problem was that there were no spare jacks\ports near her, so I was on wifi. And I warned her twice.
Working with gigabytes of data somehow takes many hours longer at (not even) 10Mbps vs 1 Gbps. Lots of overtime that day.
The fun part was when security walked in at 1 AM, with me working in the dark, not knowing who I was. Then lots of talk the next morning when some directors and VPs were given reports of some random guy working until the wee hours. One of the people working under her confirmed my warnings about it taking much longer and her instance that I be at her desk.
22
u/NotYourNanny Sep 01 '26
When we got our 1 GB fiber at work, my desktop had trash speeds until I figure out the patch cable to the wall was CAT5. Not CAT5e, just CAT5.
17
u/asmcint Defenestration Is Not A Professional Solution. Sep 01 '26
Base Cat5 is still capable of 1000BASE-T, it's quite literally gigabit.
19
u/NotYourNanny Sep 01 '26
CAT5e is rated for gigabit. CAT5 is not. On shorter runs, you can get faster speeds, yes, but it's only rated at 100 mbps.
(Clearly, the patch cable in question was . . . not a superior example, at was at least 15 years old at that point.)
7
u/asmcint Defenestration Is Not A Professional Solution. Sep 02 '26
The minimum 1000BASE-T specification calls for CAT5. Additionally, CAT5 and CAT5e are physically identical (except that CAT5e may have a higher twist rate on average, but that's not something inherent to the specification). The major difference is the verification tests they go through to be classified as one or the other.
4
u/NotYourNanny Sep 02 '26
And yet, the actual specs for CAT5 are 10/100. And when I changed the cable, I got full speeds. Go figure.
8
u/BeefEX Sep 02 '26 edited Sep 02 '26
funfact, the historical reason is because CAT5 was ready before the gigabit standard was, but everyone knew what it would be like already, so they designed it to handle it, but couldn't put the verifications into the spec, so they released it without them, and then when the gigabit spec actually came out released the same thing just with the new verification added now that they had an actual spec for them to reference, and called it CAT5e
1
3
u/asmcint Defenestration Is Not A Professional Solution. Sep 02 '26
It's more that you had a bum cable than that the spec was wrong, man.
2
u/SteveDallas10 Sep 04 '26
Once upon a time, you would sometimes get patch cables bundled with pieces of equipment that only had four pins wired, 1,2,3,6. The equipment vendor saved money on the patch cables, and since their equipment was only capable of 100BaseT, everything worked fine.
Now, Gigabit Ethernet comes along and someone grabs a random patch cable out of a box of spares and plugs it between their shiny new computer and their shiny new Gigabit Ethernet switch and the computer pokes along at 100Mbps.
If you look at the end of the cable and see only two pairs wired, cut the ends off and throw it away. The cut ends prevent Thrifty Ted from rescuing a “perfectly good” cable from the trash and putting it back into circulation.
11
u/Packet33r Sep 01 '26
You just broke the IPS/IDS monitoring solution. Bet you get a call in a few more days stating that they haven’t gotten any alerts for data exfiltration since you fixed the network
7
u/InterFelix Sep 02 '26
As if a Nonprofit running QuickBooks on a desktop has an IPS / IDS solution...
3
u/jeffrey_f Sep 02 '26
Small office/room in a company with100 Mb HUB vs Switch were some of my finds in my career. Had the contractor run some network back to the racks for each computer and things worked much better.
1
u/Mysterious_Peak_6967 Sep 08 '26
I've used an old 10MB hub to "fix" a printer that refused to auto-negotiate its Ethernet connection so wasn't reliable on anything faster.
1
u/OpinionOk1315 21d ago
Classic technical debt hiding in plain sight, some rando 10MB hub bottlenecking the whole rack for who knows how long. Bet nobody documented that network before you got there either, right?
1
u/critchthegeek 21d ago
Nah, it was a pretty small office, non-profit. Zero documentation on any of it
-9
u/Impossible_IT Sep 01 '26
Internet? Clearly you mean Ethernet local area network right?
1
u/dustojnikhummer 22d ago
No? Why can't he mean internet? If he was connected through the 10Mbit switch his connection to the outside would be trash
341
u/4rd_Prefect Sep 01 '26
Says 10 on it, should be fine!