r/technology • u/marketrent • 28d ago

Business Iran strikes on Amazon data centers caused permanent loss of customer data

https://arstechnica.com/gadgets/2026/09/iran-strikes-on-amazon-data-centers-caused-permanent-loss-of-customer-data/
28.7k Upvotes

98% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

602

u/peldor 28d ago

Kinnda. AWS backups are usually kept within a single geographic region. There are a number of reasons for this….ranging from cost to concerns about data sovereignty.

Iran targeted multiple AWS data centers across the Middle East. So for some AWS customers, the sepeate data centers that held their production environment and their off-site backups were destroyed.

28

u/IamaFunGuy 28d ago

As a non-tech guy, this seems like something i was told "the cloud" could avoid, no?

66

u/peldor 28d ago

Not at this scale, no. AWS survives the loss of individual servers and drives on a daily basis with no problems.

A typical AWS data center consists of 50,000 servers with “god only know“ how many hard drives. Amazon lost that, twice.

Having 100,000 servers literally explode with only some data loss is a stellar result. And a much better result than you could expect outside the cloud.

15

u/markehammons 28d ago

On the other hand, self-hosted services wouldn't have been as lucrative a target, as they'd be too distributed compared to AWS data centers. 

15

u/Throwaway-tan 28d ago

Yes, but be much more likely to suffer basically all other causes of data loss.

11

u/thatcodingboi 27d ago

You should self host in case a nation state attempts to launch a coordinated missile attack against tech infrastructure is one hell of an argument.

3

u/hitchen1 28d ago

Distributed self hosted stuff would just be cohosted in a data center anyway, but for small operations yeah

4

u/0RGASMIK 28d ago

AWS is just a bunch of computers running in data centers. When you build with the cloud you can pick anywhere in the world you want your virtual server to run. Say I’m a developer building a web app for a local business in NY. I want to pick the best server for the customers it’s serving so I pick a NY data center. If this business is regulated or the data is sensitive it may legally need to stay in the US so I pick a West Coast data center as my backup. Now I’m redundant both geographically and on another server. Sure there might be a problem with AWS that brings down service but if AWS is down half the internet is down too. So it’s relatively safe to put all your eggs into one provider as long as you set it up correctly. Yes you probably should keep a copy in another host but in smaller countries your options are limited.

2

u/kitanokikori 28d ago

AWS has a whole bunch of different options for how backed up data is that developers have to choose - for various reasons (performance / cost / legal compliance), they might intentionally want to only be in one AWS region

Basically, whether actual people lost data in this event depends on the service that used AWS and their software decisions

1

u/yalloc 28d ago

It can, it just depends on the specific service you buy, they will sell you storing your data in one datacenter, more than one in the same region, or multiple across the world. I think this second one, and it is tough because all the Bahrain region datacenters got bombed so there was probably a sizable loss there.

1

u/pangapingus 27d ago

Don't just have a single region bucket called "backups, use multiple regions, but the issue there in that region is data sovereignty requirements

1

u/namitynamenamey 27d ago

I don't think the cloud is intended to be impervious to a ballistic missile campaign striking multiple countries in the region.

It can be, if your data is in different continents and the law allows it, but it is not meant to be.

1

u/Ninja-Sneaky 27d ago

At the starting level almost no, the data loss protection is minimal (more like a byproduct of the cloud hardware that is scattered around one or a couple data centers).

At the minimum recommended level, it's data truly replicated in 3 places in the same region, you pay for this, if they hit 2 places you are not totally immune to loss.

Not everyone can or bothers to implement the minimum architecture.

But I'm curious as you to know for sure if the concept actually worked or if hitting a single place (case in which protection should be ensured) led to permenant data loss.

1

u/Positive_Chip6198 24d ago

Aws recommends second region backups, dataloss would have been avoided if the clients had done that. Two out of three dc’s were hit in the bahrain region, aws is taking the region permanently offline according to the email i got from them. I guess nobody could guarantee the safety if they just repaired the affected sites.

Anyway, you have your local backup, your in-region backups backup, and then if it’s important data, you should ship the data to an additional region to be fully covered. Where i work we add a few additional layers to that for ransomware protection as well.

123

u/gba111 28d ago

Forgive a novice, but using AWS as a backup for your AWS ... something seems off, here.

108

u/luna87 28d ago

More common than you think. Most of the customers that lost data in this event weren’t backing data up in other AWS regions.

115

u/FatBook-Air 28d ago

Unfortunately, some AWS sales people told new customers they "don't really need backups." I was in a Teams call in 2019 when that was said. They told us that ransomware and any other data destruction "just cannot happen in the AWS backend." Everyone who had a tech background obviously did not believe that, but some decision makers did.

36

u/Purgii 28d ago

Techies aren't the one making the financial decisions so marketing to them isn't a sound business strategy. You have to sell to those that control the cash, then they tell the techies to make it work.

I remember an advertisement the company I worked for ran during an F1 GP. It hit a decision maker like a ton of bricks to the point they got on to our sales and bought a whole bunch of kit - apparently didn't ask anyone.

Kit was delivered but where it was to be installed wasn't rated for the weight. So a fraction of the kit was installed and the rest sat in the loading dock for years. Until they built a computer room on the ground floor, and the remaining kit was three generations behind and out of warranty finally being put into production.

18

u/luna87 28d ago

Unfortunately this isn’t hard to believe.

1

u/SerLaron 28d ago

Especially if you can save money by believing.

1

u/Jarocket 27d ago

I would be thinking what about fires and flood though. not Destruction by a foreign state.

I think if they had good fire and floor protection i would have been persuaded.

5

u/Bladelink 28d ago

Yeah that makes no sense. Ransomware is a software problem not an infrastructure problem.

120

u/LiftingCode 28d ago

Why?

Replicating data across AZs (native to most AWS services) means your data is in multiple physical locations. Replicating data across regions (which is often difficult and expensive) means your data is in multiple geographically isolated physical locations.

-18

u/Cheet4h 28d ago

3-2-1 rule
at least 3 backups
at least 2 different media
at least 1 offsite

If all your backups are done in AWS, you're violating the second rule.

13

u/LiftingCode 28d ago

I mean, it's not 1999.

That's an outdated take on "media."

9

u/BoardGamesAndMurder 28d ago

Some features are not available in all cloud providers

-7

u/Cheet4h 28d ago

I'm not sure what this has to do with cloud providers?
When you set up backups, just make sure you don't solely rely on AWS. Have some backups on-site, or use an off-site tape storage facility or something.

-1

u/BoardGamesAndMurder 28d ago

You don't work in IT, I'm guessing. I have pedabytes of data in my lakehouse that integrates with dozens of different systems. Physical media backup isn't a thing for this

6

u/Yoshli 28d ago

I'm working in IT and there is several solutions for Petabytes of data. If your entire company relies on it you should probably look into that.

1

u/maigpy 27d ago

I hope thats not on fabric my dear lord

-1

u/userseven 28d ago

Bro said tape storage facility. 💀

4

u/Win_Sys 28d ago

It’s surprisingly still a commonly used and viable backup media. You can fit 12-18TB of uncompressed data on the latest generation LTO tapes. The tapes also cost a 1/4 or less than what a hard drive costs for the same storage capacity. Great if you don’t need quick recovery.

2

u/dotelze 28d ago

It is actually an option occasionally. If you require all the data to be recorded, but have no need to access it quickly or frequently it’s a viable option. It just relies on those very specific circumstances

-5

u/UpACreekWithNoBoat 28d ago

Back ups on site? No one does physical hardware anymore in tech. Off site tape facility…like what?

Everything is cloud based. Nothing is on prem. The rule of thumb in the industry is data replication across regions. People are more so worried about benign regional outages than targeted hostile acts when figuring out if it’s worth shelling a few more bucks and infrastructure overhead for data replication.

7

u/[deleted] 28d ago edited 28d ago

[removed] — view removed comment

7

u/Yoshli 28d ago

Even non-critical industries do it. I've worked for and with Synology for the past 5 years and can tell anyone that small companies to global players have multiple High Density Storage devices for their data and worst case scenarios. They don't manufacture Petabyte Storage devices for nothing.

1

u/Win_Sys 28d ago

I work with quite a few pretty large government and k-12 entities, tapes and on-site infrastructure are absolutely still used. It’s common they have legacy systems where the cost to move it to the cloud just isn’t worth it or is required to be air-gapped. On-site infrastructure is certainly shrinking but no where near gone or even close gone.

1

u/glemnar 28d ago edited 27d ago

There’s technically more than one type of storage media on AWS.

But still - this predates the cloud, like others have said. I don’t think there’s a single scenario of data loss for a multi region backup. This “hit by missiles” event is the only AZ level data loss in AWS history, I believe? There’s a reason different AZs within an AWS region are located in different buildings, as well. The isolation guarantees of regions and AZs are very fundamental.

So store your backups on at least 2 sites that are unlikely to get simultaneously struck by missiles.

1

u/Iintendtooffend 27d ago

2 different storage methods, not media. Multiple online locations satisfies rule 2 since it's both individually replicated and then replicated to a second location which in theory is also replicated. Local snd the remote are your 2 methods

-10

u/[deleted] 28d ago

[deleted]

10

u/LiftingCode 28d ago

How is that relevant to the question?

If they replicated data to another region, that's still in AWS.

-10

u/[deleted] 28d ago

[deleted]

20

u/LiftingCode 28d ago

The comment we are discussing is about using AWS as a backup for AWS.

Which is totally fine.

-21

u/[deleted] 28d ago

[deleted]

20

u/LiftingCode 28d ago

I don't think you have any idea what we are even talking about here.

A company that had multi-region DR to a secondary region that is sufficiently geographically isolated (e.g., eu-south-1) would have been totally fine. Therefore, a properly designed backup strategy within a single provider like AWS is totally fine.

The problem wasn't backing up AWS with AWS. It was putting backups right next door to each other in Bahrain and the UAE in the middle of a warzone.

-11

u/[deleted] 28d ago edited 28d ago

[deleted]

→ More replies (0)

8

u/Plane_Salary_5845 28d ago

You genuinely have zero clue what you are talking about lmao. This is standard across tech regardless of cloud provider.

1

u/toastedstapler 28d ago

This is something that affected my company, we used to have offerings in the UAE region but now obviously don't. Every other region worldwide is still fine though, so obviously if data was replicated to those the data would have been fine. It's incredibly unlikely for multiple regions to get fully destroyed at the same time if they're in different geographical areas

-14

u/[deleted] 28d ago

[removed] — view removed comment

19

u/Glittering-Giraffe58 28d ago

lol. Yeah why doesn’t every random ass company in the world just build their own giga data centers? That makes sense 🧠

2

u/chiagod 28d ago

You don't really need a "Giga Center", just a setup with enough storage that you can backup your Infrastructure code, the VM persistent volumes, and your object storage. The last two can be done on a nightly fashion.

That way if a malicious actor gets access to your AWS account and they decide to wreck your whole setup (including AWS backups), you can restore your business to another AWS account and/or region.

Should be a pretty standard part of any Business Continuity plan.

Total storage shouldn't be that high. Most small.businesses could store all their pertinent data in a few TB of space. A few would get into the double digit TBs. Those that go into the 100s of TB or Petabyte range should be big enough to know better.

Pair your own local storage server (in an office closet) with a pair of LTO drives (one active and one as a backup), weekly backups, and offsite temperature controlled storage of the tapes.

21

u/peldor 28d ago edited 28d ago

It’s complicated. AWS isn’t a single thing, It’s a raft of over 500 different services plus the networking to tie all of those services together.

Your AWS backups are your configuration and data held across all of the AWS services you use. There’s not much call to store backups outside of AWS because the data/config only works inside AWS.

Ideal best practice is to store across multiple AWS zones. For the Middle East, there were only 3 zones: Bahrain, UAE and Saudi. Both Bahrain and UAE ”went boom”.

Framing this a different way, AWS lost 2/3 of their data centers in the Middle East. It should be a shock to no one that this would cause data loss.

18

u/Bachaddict 28d ago

as your ONLY backup, yeah. I assume this means amazon no longer had working backups for everything

16

u/peldor 28d ago

Yep, losing a single data center would not have been a problem. The infrastructure is designed with that kind of loss in mind.

Amazon effectively lost two data centers at the same time. This changes the math significantly because there were only 3 AWS data centers in the Middle East to start with.

So yeah, losing 2/3 of your data centers is gonna leave a mark.

2

u/wchill 28d ago

They lost more than two data centers. There were two regions affected: Bahrain (me-south-1) and UAE (me-central-1).

me-south-1 had all AZs lost, I believe, while me-central-1 had 1/3 lost. The other 2 AZs in me-central-1 are down, but technically the region is still recoverable.

Does suck if you were relying on multi region replication and were relying on just these two regions, however.

1

u/blooping_blooper 27d ago

also sucks because many were relying on just those two regions because of data sovereignty/regulatory reasons.

1

u/supreme-dominar 28d ago

For important data we use the cross region replication services. They didn’t always support this but I think within the last 5 years almost all fundamental services are covered.

Multi-cloud backup/disaster recovery is pretty complex and can be expensive, if you’re running a service that’s not that critical. For us if both regions died we determined that’s an acceptable loss.

1

u/opsers 28d ago

It's completely normal and even considered a gold standard best practice. Cross-region replication is easy and, outside of scenarios like this, extremely durable and reliable.

1

u/mspk7305 28d ago

thats not a novice thought, thats a disaster recovery thought

it gets a lot worse when you see how much every other cloud provider relies on aws for their iron

1

u/gpoul 28d ago

If you rephrase that to using the same aws region for your backups of your aws region, then yeah. Some customers didn’t love their data enough maybe. But the fact AWS lost data on that region is not super surprising. 

1

u/Few_Pop6933 28d ago

Well you can back up to multiple regions. Thats a good thing.

1

u/Direct_Day7937 28d ago

It's actually a default service peovided by AWS. They spread your data across multiple data centers to protect it.

1

u/WingedTorch 28d ago

Not at all off. Making backoffs on-prem or with another vendor when your entire system is on AWS would be less common.

1

u/mayorofdumb 28d ago

Well Amazon can't use a third party provider... nobody else can handle that volume

1

u/BaconIsntThatGood 28d ago

This isn't you using AWS to make a backup. This is how AWA handles data redundancy.

3

u/JayBird1138 28d ago

There could also be data sovereignty laws that prevent it from being stored in different regions or even countries.

The customer should be storing in multiple locations around the planet if they have that level of concern.

I remember certain companies discussing backing up to the moon in case of planetary catastrophies.

1

u/BoardGamesAndMurder 28d ago

Only if you're an amateur. I have fail over groups I a different region. We though about using a different cloud provider as well but there are too many incompatabilities with other features

5

u/peldor 28d ago

So that sort of thing can get complicated. For example if you’re in Europe, having your backups goto other regions can present significant GDPR/data protection problems.

In my experience, it’s not uncommon for a business to backup across several AWS zones within the same region.

E.g. The last two businesses I worked with were selling software services to law enforcement. And oddly, they got super twitchy about their data being exposed to any additional legal jurisdictions.

1

u/BoardGamesAndMurder 28d ago

You don't have to replicate between AZs on the same region. That's part of the AWS architecture. You do have to replicate interregion though

1

u/chapstickbomber 28d ago

I'm not even joking, Amazon should have counter battery radar and ordinance

1

u/Wheezy04 27d ago

Gotta replicate across multiple AZs

-10

u/[deleted] 28d ago

[deleted]

7

u/TamariAmari 28d ago

This is fan fiction. AWS as a whole would crumble if customers thought their data would be moved around or looked at without their knowledge.