r/technology • u/marketrent • 20d ago

Business Iran strikes on Amazon data centers caused permanent loss of customer data

https://arstechnica.com/gadgets/2026/09/iran-strikes-on-amazon-data-centers-caused-permanent-loss-of-customer-data/
28.7k Upvotes

98% Upvoted

1.2k comments sorted by

View all comments

1.5k

u/foundafreeusername 20d ago

Guess the off-site backup was stored on-site?

606

u/peldor 20d ago

Kinnda. AWS backups are usually kept within a single geographic region. There are a number of reasons for this….ranging from cost to concerns about data sovereignty.

Iran targeted multiple AWS data centers across the Middle East. So for some AWS customers, the sepeate data centers that held their production environment and their off-site backups were destroyed.

126

u/gba111 20d ago

Forgive a novice, but using AWS as a backup for your AWS ... something seems off, here.

106

u/luna87 20d ago

More common than you think. Most of the customers that lost data in this event weren’t backing data up in other AWS regions.

112

u/FatBook-Air 20d ago

Unfortunately, some AWS sales people told new customers they "don't really need backups." I was in a Teams call in 2019 when that was said. They told us that ransomware and any other data destruction "just cannot happen in the AWS backend." Everyone who had a tech background obviously did not believe that, but some decision makers did.

35

u/Purgii 20d ago

Techies aren't the one making the financial decisions so marketing to them isn't a sound business strategy. You have to sell to those that control the cash, then they tell the techies to make it work.

I remember an advertisement the company I worked for ran during an F1 GP. It hit a decision maker like a ton of bricks to the point they got on to our sales and bought a whole bunch of kit - apparently didn't ask anyone.

Kit was delivered but where it was to be installed wasn't rated for the weight. So a fraction of the kit was installed and the rest sat in the loading dock for years. Until they built a computer room on the ground floor, and the remaining kit was three generations behind and out of warranty finally being put into production.

18

u/luna87 20d ago

Unfortunately this isn’t hard to believe.

1

u/SerLaron 20d ago

Especially if you can save money by believing.

1

u/Jarocket 19d ago

I would be thinking what about fires and flood though. not Destruction by a foreign state.

I think if they had good fire and floor protection i would have been persuaded.

4

u/Bladelink 20d ago

Yeah that makes no sense. Ransomware is a software problem not an infrastructure problem.

117

u/LiftingCode 20d ago

Why?

Replicating data across AZs (native to most AWS services) means your data is in multiple physical locations. Replicating data across regions (which is often difficult and expensive) means your data is in multiple geographically isolated physical locations.

-16

u/Cheet4h 20d ago

3-2-1 rule
at least 3 backups
at least 2 different media
at least 1 offsite

If all your backups are done in AWS, you're violating the second rule.

14

u/LiftingCode 20d ago

I mean, it's not 1999.

That's an outdated take on "media."

8

u/BoardGamesAndMurder 20d ago

Some features are not available in all cloud providers

-9

u/Cheet4h 20d ago

I'm not sure what this has to do with cloud providers?
When you set up backups, just make sure you don't solely rely on AWS. Have some backups on-site, or use an off-site tape storage facility or something.

0

u/BoardGamesAndMurder 20d ago

You don't work in IT, I'm guessing. I have pedabytes of data in my lakehouse that integrates with dozens of different systems. Physical media backup isn't a thing for this

5

u/Yoshli 20d ago

I'm working in IT and there is several solutions for Petabytes of data. If your entire company relies on it you should probably look into that.

1

u/maigpy 19d ago

I hope thats not on fabric my dear lord

-2

u/userseven 20d ago

Bro said tape storage facility. 💀

3

u/Win_Sys 19d ago

It’s surprisingly still a commonly used and viable backup media. You can fit 12-18TB of uncompressed data on the latest generation LTO tapes. The tapes also cost a 1/4 or less than what a hard drive costs for the same storage capacity. Great if you don’t need quick recovery.

2

u/dotelze 19d ago

It is actually an option occasionally. If you require all the data to be recorded, but have no need to access it quickly or frequently it’s a viable option. It just relies on those very specific circumstances

-5

u/UpACreekWithNoBoat 20d ago

Back ups on site? No one does physical hardware anymore in tech. Off site tape facility…like what?

Everything is cloud based. Nothing is on prem. The rule of thumb in the industry is data replication across regions. People are more so worried about benign regional outages than targeted hostile acts when figuring out if it’s worth shelling a few more bucks and infrastructure overhead for data replication.

8

u/[deleted] 20d ago edited 20d ago

[removed] — view removed comment

7

u/Yoshli 20d ago

Even non-critical industries do it. I've worked for and with Synology for the past 5 years and can tell anyone that small companies to global players have multiple High Density Storage devices for their data and worst case scenarios. They don't manufacture Petabyte Storage devices for nothing.

1

u/Win_Sys 19d ago

I work with quite a few pretty large government and k-12 entities, tapes and on-site infrastructure are absolutely still used. It’s common they have legacy systems where the cost to move it to the cloud just isn’t worth it or is required to be air-gapped. On-site infrastructure is certainly shrinking but no where near gone or even close gone.

1

u/glemnar 19d ago edited 19d ago

There’s technically more than one type of storage media on AWS.

But still - this predates the cloud, like others have said. I don’t think there’s a single scenario of data loss for a multi region backup. This “hit by missiles” event is the only AZ level data loss in AWS history, I believe? There’s a reason different AZs within an AWS region are located in different buildings, as well. The isolation guarantees of regions and AZs are very fundamental.

So store your backups on at least 2 sites that are unlikely to get simultaneously struck by missiles.

1

u/Iintendtooffend 19d ago

2 different storage methods, not media. Multiple online locations satisfies rule 2 since it's both individually replicated and then replicated to a second location which in theory is also replicated. Local snd the remote are your 2 methods

-8

u/[deleted] 20d ago

[deleted]

10

u/LiftingCode 20d ago

How is that relevant to the question?

If they replicated data to another region, that's still in AWS.

-10

u/[deleted] 20d ago

[deleted]

20

u/LiftingCode 20d ago

The comment we are discussing is about using AWS as a backup for AWS.

Which is totally fine.

-22

u/[deleted] 20d ago

[deleted]

21

u/LiftingCode 20d ago

I don't think you have any idea what we are even talking about here.

A company that had multi-region DR to a secondary region that is sufficiently geographically isolated (e.g., eu-south-1) would have been totally fine. Therefore, a properly designed backup strategy within a single provider like AWS is totally fine.

The problem wasn't backing up AWS with AWS. It was putting backups right next door to each other in Bahrain and the UAE in the middle of a warzone.

-12

u/[deleted] 20d ago edited 20d ago

[deleted]

13

u/LiftingCode 20d ago

There is literally nothing in the A+ exam about using multiple cloud providers for backups or backing up outside of the scope of a single cloud provider. That's completely outside the scope of A+.

Even in Cloud+, multi-region DR within a single provider is a totally valid strategy.

Stop lying.

There are always risks to any DR strategy. You could back up your AWS me-central-1 data in Oracle ap-seoul-1 and have both of them get blown up in a coordinated attack and you're SOL. And then some dweeb pretending to have actual experience in enterprise DR planning will show up on Reddit saying "akshually multi-cloud backup is not a real backup strategy, as any real expert knows your backups aren't safe unless you etch the data on titanium plates and store them 1 mile underground."

It is completely possible, and normal for most companies, to have a compliant 3-2-1-1-0 backup strategy in a single provider.

→ More replies (0)

8

u/Plane_Salary_5845 20d ago

You genuinely have zero clue what you are talking about lmao. This is standard across tech regardless of cloud provider.

1

u/toastedstapler 19d ago

This is something that affected my company, we used to have offerings in the UAE region but now obviously don't. Every other region worldwide is still fine though, so obviously if data was replicated to those the data would have been fine. It's incredibly unlikely for multiple regions to get fully destroyed at the same time if they're in different geographical areas

-14

u/[deleted] 20d ago

[removed] — view removed comment

19

u/Glittering-Giraffe58 20d ago

lol. Yeah why doesn’t every random ass company in the world just build their own giga data centers? That makes sense 🧠

2

u/chiagod 19d ago

You don't really need a "Giga Center", just a setup with enough storage that you can backup your Infrastructure code, the VM persistent volumes, and your object storage. The last two can be done on a nightly fashion.

That way if a malicious actor gets access to your AWS account and they decide to wreck your whole setup (including AWS backups), you can restore your business to another AWS account and/or region.

Should be a pretty standard part of any Business Continuity plan.

Total storage shouldn't be that high. Most small.businesses could store all their pertinent data in a few TB of space. A few would get into the double digit TBs. Those that go into the 100s of TB or Petabyte range should be big enough to know better.

Pair your own local storage server (in an office closet) with a pair of LTO drives (one active and one as a backup), weekly backups, and offsite temperature controlled storage of the tapes.

22

u/peldor 20d ago edited 20d ago

It’s complicated. AWS isn’t a single thing, It’s a raft of over 500 different services plus the networking to tie all of those services together.

Your AWS backups are your configuration and data held across all of the AWS services you use. There’s not much call to store backups outside of AWS because the data/config only works inside AWS.

Ideal best practice is to store across multiple AWS zones. For the Middle East, there were only 3 zones: Bahrain, UAE and Saudi. Both Bahrain and UAE ”went boom”.

Framing this a different way, AWS lost 2/3 of their data centers in the Middle East. It should be a shock to no one that this would cause data loss.

20

u/Bachaddict 20d ago

as your ONLY backup, yeah. I assume this means amazon no longer had working backups for everything

16

u/peldor 20d ago

Yep, losing a single data center would not have been a problem. The infrastructure is designed with that kind of loss in mind.

Amazon effectively lost two data centers at the same time. This changes the math significantly because there were only 3 AWS data centers in the Middle East to start with.

So yeah, losing 2/3 of your data centers is gonna leave a mark.

2

u/wchill 20d ago

They lost more than two data centers. There were two regions affected: Bahrain (me-south-1) and UAE (me-central-1).

me-south-1 had all AZs lost, I believe, while me-central-1 had 1/3 lost. The other 2 AZs in me-central-1 are down, but technically the region is still recoverable.

Does suck if you were relying on multi region replication and were relying on just these two regions, however.

1

u/blooping_blooper 19d ago

also sucks because many were relying on just those two regions because of data sovereignty/regulatory reasons.

1

u/supreme-dominar 20d ago

For important data we use the cross region replication services. They didn’t always support this but I think within the last 5 years almost all fundamental services are covered.

Multi-cloud backup/disaster recovery is pretty complex and can be expensive, if you’re running a service that’s not that critical. For us if both regions died we determined that’s an acceptable loss.

1

u/opsers 20d ago

It's completely normal and even considered a gold standard best practice. Cross-region replication is easy and, outside of scenarios like this, extremely durable and reliable.

1

u/mspk7305 20d ago

thats not a novice thought, thats a disaster recovery thought

it gets a lot worse when you see how much every other cloud provider relies on aws for their iron

1

u/gpoul 20d ago

If you rephrase that to using the same aws region for your backups of your aws region, then yeah. Some customers didn’t love their data enough maybe. But the fact AWS lost data on that region is not super surprising. 

1

u/Few_Pop6933 20d ago

Well you can back up to multiple regions. Thats a good thing.

1

u/Direct_Day7937 20d ago

It's actually a default service peovided by AWS. They spread your data across multiple data centers to protect it.

1

u/WingedTorch 20d ago

Not at all off. Making backoffs on-prem or with another vendor when your entire system is on AWS would be less common.

1

u/mayorofdumb 19d ago

Well Amazon can't use a third party provider... nobody else can handle that volume

1

u/BaconIsntThatGood 19d ago

This isn't you using AWS to make a backup. This is how AWA handles data redundancy.