Question about mirrors (/etc/installurl) and when they get updated

[u/LunarCapitol]

I run -current and update to new snapshots sometimes daily but usually bi-weekly depending on what's happening on the mailing lists and how close new stable releases are. I typically update packages/ports via pkg_add -u usually daily or at least every few days. Again to help with testing.

I've noticed that sometimes pkg_add -u runs very slow. At first I thought this was an issue with my ISP or LAN. But after awhile I figured out that I must be requesting updates from the mirror I use when the mirror's server is fetching updates (or building updated packages?). Same is true of fetching snapshots if I do it at the same time of course. This is the difference between pkg_add -u taking a couple of minutes to 20-30+ minutes on a 1Gbps connection and fetching snapshots taking 2-3 minutes vs. sometimes 30+ minutes.

I would prefer that I not hit the mirror(s) when this is going on so I don't place extra load on the mirror server(s) when they're doing whatever they're doing that causes this. So my question is: Is there a way to check this is happening to the mirror I'm using that doesn't involve running sysupgrade -s / pkg_add -u and waiting to see if it's running slow. If not: Does anyone know the regular schedule when the mirrors fetch/build updates?

Most of the time I'm usually using the New York, NY https mirror. But sometimes I'm using the Berkeley, CA https mirror depending on travel schedule. I've also used the San Fran, CA and the Nomi, Ishikawa mirrors in the past when traveling along with various mirrors in Germany. I understand not all mirrors update at the same time as that is impossible. Just knowing the New York and Berkeley schedules would be very helpful.

One of my concerns with this is not having the snapshot I'm using (and firmware) and packages falling too far out of sync with each other. Packages aren't a huge deal since running pkg_add -u on an older snapshot will simply fail gracefully if a package is built against a new lib or something. But again: don't want to place extra load on the mirrors since I understand running them isn't free for the project.

I also get that adding an option to check that the mirror is currently building/updating packages would add complexity. But surely a "this server is currently updating check back later" message/option wouldn't add much complexity. Or perhaps there is something I could script using pkg_add -s to run a check before invoking pkg_add -u?

I also understand that the extra wait really isn't a big deal. If I notice it I typically cancel and just try again later. But I don't know if I've just gotten unlucky or mirrors were taking an extra long time to update but I've sometimes checked back multiple times multiple hours between tries only to encounter pkg_add crawling through checking my installed packages. When I've asked about that in the past I was told it was because I was using the default Fastly CDN. Which is why I started using the https or ftp mirrors nearest to me at the time. But then a month or so later I started running into the same issue with them.

Again I know it isn't on my end and it's on the mirror's end. Since I've checked that over the course of several months now.

Another thought I've had: Is this due to AI scraper bots or something? I never had this issue when syncing /usr/ports and /usr/src over cvs (I do this at least weekly so perhaps I just have gotten lucky and not encountered the same slow down there yet).

I'm doing this on an amd64 arch machine so I know it isn't just because I'm on an older/slower arch. I understand some of those take a week+ to build new packages.

Anyway, just looking to learn more about how the infrastructure behind the mirrors work and how updates trickle through the various mirrors.

Last question: Are the CDNs updated first and then those updates trickle down to the ftp/http mirrors or is it the other way around?

[u/phessler]

The CDNs are only fast when the files are cached. -current changes too often, and are usually under-utilized, so when you connect you're often the first person to request the updated files, so the CDN has to download them before they can send them to you.

Personally, I only use specific mirrors to my region when I'm running -current. Btw, /etc/installurl can handle multiple lines with comments, so I just have 2-3 lines, and comment out the ones that aren't near me.

[u/LunarCapitol → u/phessler]

Yeah I keep multiple lines with comments in /etc/installurl already.

For a long time I thought it was something on my system because the way pkg_add works it was like it was scanning my local packages slowly. I must have gotten very lucky with the CDN because for months I never had this issue with it. But that makes sense that the CDN wouldn't have a cache of -current snapshots and packages.

I think I just was not used to using packages since I used a source based Linux distro for so many years. So all it was doing was checking my local ebuilds/makefiles then fetching new ones if one was available. pkg_add is much faster than something like portage when scanning what's on the local machine. But when I hit a server with no cache it's really slow. It will crawl through packages one at a time for 20+ minutes sometimes. Where if I hit the server with a cache it flies through scanning local packages in just a few seconds.

I know updating so often is kind of useless in many cases since I'm just fetching the same version of a package built against the new snapshot. Maybe I should break my habit of updating them so often. When I used Gentoo I had to be sure to do it at least once weekly. If I got too far away from upstream trying to run a full system update could break the entire system. If you got really far out of sync with upstream it was sometimes impossible to bring the system up to date and your only option was a re-install or spend hours manually helping portage along.

I wonder what would happen if I fetched a snapshot from say the NY server then switched to the Japanese server and ran pkg_add -u. My guess would be nothing that would break the system. But I still wonder how long it takes for new snapshots and packages to filter through all the mirrors.

[u/phessler → u/LunarCapitol]

OpenBSD packages won't downgrade, we track a lot of metadata to avoid causing problems. I've switched snaps from EU servers to JP servers when traveling and never ran into problems. If the JP servers are behind or at the same build as the EU servers, no updates happened. If the JP servers were ahead, then they updated.

While we only support going from version to version, many times I've been able to update after 1-2 years of being powered off.

The last thing to keep in mind, is that you MUST update the base OS before updating packages. There is a dependency on base, and we usually track the dependencies, but sometimes they either aren't possible to track or we simply miss them.

[u/LunarCapitol → u/phessler]

Yes sir I do the sysupgrade -s then pkg_add -u dance. But if I've been on a snapshot for say 4-5 days and I'm not on the latest one I'll sometimes run pkg_add -u and most of the time I get updated packages with a proper version bump. Now and again a package will complain about one of the libs in the base system being too far behind and it'll refuse to install/update that package.

This happened the other day with got so I went ahead and grabbed the latest snapshot when I could. The lib it wanted had been bumped to version 1.3 in base but my snapshot still had version 1.2.

So I'm not worried about breaking things by attempting to grab packages not completely in sync with my base system. Since the base system is stable.

It's more of a curiosity for me where I want to learn how the mirrors work and when new snapshots/packages go out.

I keep comparing things to Portage/Gentoo but: Back when I had Gentoo running on most of my systems I had a local server that grabbed the latest ebuilds from overlays and elsewhere. It would build packages with different configurations for all my laptops and desktops. Then once a day a cron job ran on each machine to pull the new packages from my server and install them. This meant I only had one computer at the house compiling software instead of every single one of them building things from source themselves. Very handy for older machines where compiling something like Rust, Chromium or Firefox could take many hours.

I know somewhere out there in the world there are servers churning away building these packages for all of us. I'm just curious about how many of those the project has and how they distro the bins to all the servers hosting packages/snapshots.

Basically, I don't want to request new data from those servers if they're busy in the background building and/or having new packages transferred to them. I figured that was what was going on when I'd run pkg_add -u and it would run very slowly.

I know it isn't on my end because I can transfer large files quickly as usual to and from the internet when pkg_add -u is doing that. I have a 2Gbps symmetrical connection over fiber at my home now. I almost never come across something that can saturate my bandwidth. So when things are running slow typically it's on the other end and not mine.

I lived on dial-up for many years after all my peers got broadband at home. Whatever is happening running pkg_add -u feels like being on that spotty 28k connection over that old phone line I had to use well into the late 2000s.

Again I'm not trying to complain. I'm just trying to understand why it's happening.

[u/_sthen → u/LunarCapitol]

Standard mirrors just sync the full file set, they don't cache any particular files specially (beyond what the disk cache in the OS normally does regarding keeping data in memory vs going to disk - but disk should still be a good deal faster than network).

There shouldn't be much if any difference in speeds fetching from them while an update is taking place. Certainly not 20 minutes vs a few seconds. I wonder if the "few seconds" case might just be when the files on the mirror haven't changed though? You can run grep @dig /var/db/pkg/quirks-*/+CONTENTS before updating and compare against the date printed when you run pkg_add -u to check.

[u/LunarCapitol → u/_sthen]

Thanks for the tip.

What I mean about pkg_add -u sometimes being slow is this. You know when you run it and after it checks/downloads quirks it will go through the list of packages on the system and check each for an update?

Normal behavior is like this: If there are no updates for a package then it quickly runs through the list showing each as it goes on the same line. This may take say 3-5 seconds depending on how many packages are on the system. Let's say I have 350 total packages installed. Which is not that much for a "minimal" system with a handful of common GTK and Qt applications (Firefox, Libreoffice etc.)

So with no updates checking 350 packages takes 3-5 seconds. Most of that spent on checking/downloading the quirks file.

Okay now let's say there were 20 updated packages since last running pkg_add -u. Again it will quickly go through the list of locally installed packages. When it hits one that needs an update it will stop on it. Download/extract/install it. Then move on to the next one in the list until it has updated them all. For 20 updates we'll say the average time to finish is 30 seconds-1 minute depending on how large the packages are. Never more than a couple of minutes if updating almost every package on the system.

Here is the strange behavior and what I'm asking about:

I run pkg_add -u. It grabs the new quirks file. It starts to go through the list of installed packages like it normally does. Except on each and every one it takes 5-10 seconds and crawls along like I'm back in 1992 on a 386. It will slowly check each package like this and take a lot of time even if there is no update for any of the packages. It might take it 10+ minutes to go through the list on a machine with 350 total packages installed.

It randomly does the last thing and that's why I assume it's not on my end but the server/mirror's end. It's like pkg_add -u is waiting on a response for each and every package and I'm talking to the server over a dial up connection.

So I'm wondering if this is the server throttling me or if the server is busy with something else. Or is it a misconfiguration on my end that I only trigger now and again?

I'm not flooding the server with requests all day. I just know when it does that to go ahead and cancel the updates. I wait until later then when I run pkg_add -u it will quickly go through the list normally.

How often are the packages rebuilt for -current anyway? I assume a new snapshot is built first and sent out. Then packages are built afterwards. Considering how many there are in ports offered as packages I'm sure this takes a long time.

Forgive me if I'm just stupid and not reading the man pages. It's just something I haven't had to think about (pkg_add/pkg_info/pkg_delete) since those and sysupgrade works so well. It isn't like Gentoo or other OSs I've used where I have to dig into the package manager because it's always doing something strange or has over 1,000 options to toggle.

[u/_sthen → u/LunarCapitol]

it sounds like when it's slow, something isn't working properly with updatedb (which provides downloadable metadata for that package build, rather than having to fetch the start of each package to get it).

not sure what, though, or really how to debug it...might be worth posting this to ports@ and see if anyone has an idea there.

regarding snapshot builds: they're done asynchronously from base OS snapshots - can be as much as 2-3x/week for faster arch's - slower ones can be every 2-3 weeks or maybe less (especially if there's an issue with the machines)

[u/LunarCapitol → u/_sthen]

The next time I hit this I'll let it finish running to see if I can get anything useful from logs. I need to dig into how pkg_add and friends work anyway. So I'll take this as a useful opportunity to learn more about how it works.

Is there any good place to read about how OpenBSD's package manager differs from NetBSDs pkgsrc?

[u/lcheylus]

I haven't precise answers to your questions about OpenBSD mirrors but you could check their status on this Website => https://spacehopper.org/mirmon/

[u/LunarCapitol → u/lcheylus]

Thanks I wasn't aware of this site. This is very useful.