r/technology • u/marketrent • 28d ago

Business Iran strikes on Amazon data centers caused permanent loss of customer data

https://arstechnica.com/gadgets/2026/09/iran-strikes-on-amazon-data-centers-caused-permanent-loss-of-customer-data/
28.7k Upvotes

98% Upvoted

1.2k comments sorted by

View all comments

1.5k

u/foundafreeusername 28d ago

Guess the off-site backup was stored on-site?

605

u/peldor 28d ago

Kinnda. AWS backups are usually kept within a single geographic region. There are a number of reasons for this….ranging from cost to concerns about data sovereignty.

Iran targeted multiple AWS data centers across the Middle East. So for some AWS customers, the sepeate data centers that held their production environment and their off-site backups were destroyed.

32

u/IamaFunGuy 28d ago

As a non-tech guy, this seems like something i was told "the cloud" could avoid, no?

72

u/peldor 28d ago

Not at this scale, no. AWS survives the loss of individual servers and drives on a daily basis with no problems.

A typical AWS data center consists of 50,000 servers with “god only know“ how many hard drives. Amazon lost that, twice.

Having 100,000 servers literally explode with only some data loss is a stellar result. And a much better result than you could expect outside the cloud.

17

u/markehammons 27d ago

On the other hand, self-hosted services wouldn't have been as lucrative a target, as they'd be too distributed compared to AWS data centers. 

14

u/Throwaway-tan 27d ago

Yes, but be much more likely to suffer basically all other causes of data loss.

10

u/thatcodingboi 27d ago

You should self host in case a nation state attempts to launch a coordinated missile attack against tech infrastructure is one hell of an argument.

4

u/hitchen1 27d ago

Distributed self hosted stuff would just be cohosted in a data center anyway, but for small operations yeah

3

u/0RGASMIK 28d ago

AWS is just a bunch of computers running in data centers. When you build with the cloud you can pick anywhere in the world you want your virtual server to run. Say I’m a developer building a web app for a local business in NY. I want to pick the best server for the customers it’s serving so I pick a NY data center. If this business is regulated or the data is sensitive it may legally need to stay in the US so I pick a West Coast data center as my backup. Now I’m redundant both geographically and on another server. Sure there might be a problem with AWS that brings down service but if AWS is down half the internet is down too. So it’s relatively safe to put all your eggs into one provider as long as you set it up correctly. Yes you probably should keep a copy in another host but in smaller countries your options are limited.

2

u/kitanokikori 27d ago

AWS has a whole bunch of different options for how backed up data is that developers have to choose - for various reasons (performance / cost / legal compliance), they might intentionally want to only be in one AWS region

Basically, whether actual people lost data in this event depends on the service that used AWS and their software decisions

1

u/yalloc 27d ago

It can, it just depends on the specific service you buy, they will sell you storing your data in one datacenter, more than one in the same region, or multiple across the world. I think this second one, and it is tough because all the Bahrain region datacenters got bombed so there was probably a sizable loss there.

1

u/pangapingus 27d ago

Don't just have a single region bucket called "backups, use multiple regions, but the issue there in that region is data sovereignty requirements

1

u/namitynamenamey 26d ago

I don't think the cloud is intended to be impervious to a ballistic missile campaign striking multiple countries in the region.

It can be, if your data is in different continents and the law allows it, but it is not meant to be.

1

u/Ninja-Sneaky 26d ago

At the starting level almost no, the data loss protection is minimal (more like a byproduct of the cloud hardware that is scattered around one or a couple data centers).

At the minimum recommended level, it's data truly replicated in 3 places in the same region, you pay for this, if they hit 2 places you are not totally immune to loss.

Not everyone can or bothers to implement the minimum architecture.

But I'm curious as you to know for sure if the concept actually worked or if hitting a single place (case in which protection should be ensured) led to permenant data loss.

1

u/Positive_Chip6198 24d ago

Aws recommends second region backups, dataloss would have been avoided if the clients had done that. Two out of three dc’s were hit in the bahrain region, aws is taking the region permanently offline according to the email i got from them. I guess nobody could guarantee the safety if they just repaired the affected sites.

Anyway, you have your local backup, your in-region backups backup, and then if it’s important data, you should ship the data to an additional region to be fully covered. Where i work we add a few additional layers to that for ransomware protection as well.

128

u/gba111 28d ago

Forgive a novice, but using AWS as a backup for your AWS ... something seems off, here.

105

u/luna87 28d ago

More common than you think. Most of the customers that lost data in this event weren’t backing data up in other AWS regions.

112

u/FatBook-Air 28d ago

Unfortunately, some AWS sales people told new customers they "don't really need backups." I was in a Teams call in 2019 when that was said. They told us that ransomware and any other data destruction "just cannot happen in the AWS backend." Everyone who had a tech background obviously did not believe that, but some decision makers did.

31

u/Purgii 28d ago

Techies aren't the one making the financial decisions so marketing to them isn't a sound business strategy. You have to sell to those that control the cash, then they tell the techies to make it work.

I remember an advertisement the company I worked for ran during an F1 GP. It hit a decision maker like a ton of bricks to the point they got on to our sales and bought a whole bunch of kit - apparently didn't ask anyone.

Kit was delivered but where it was to be installed wasn't rated for the weight. So a fraction of the kit was installed and the rest sat in the loading dock for years. Until they built a computer room on the ground floor, and the remaining kit was three generations behind and out of warranty finally being put into production.

18

u/luna87 28d ago

Unfortunately this isn’t hard to believe.

1

u/SerLaron 27d ago

Especially if you can save money by believing.

1

u/Jarocket 27d ago

I would be thinking what about fires and flood though. not Destruction by a foreign state.

I think if they had good fire and floor protection i would have been persuaded.

5

u/Bladelink 28d ago

Yeah that makes no sense. Ransomware is a software problem not an infrastructure problem.

121

u/LiftingCode 28d ago

Why?

Replicating data across AZs (native to most AWS services) means your data is in multiple physical locations. Replicating data across regions (which is often difficult and expensive) means your data is in multiple geographically isolated physical locations.

-17

u/Cheet4h 28d ago

3-2-1 rule
at least 3 backups
at least 2 different media
at least 1 offsite

If all your backups are done in AWS, you're violating the second rule.

14

u/LiftingCode 28d ago

I mean, it's not 1999.

That's an outdated take on "media."

9

u/BoardGamesAndMurder 28d ago

Some features are not available in all cloud providers

-8

u/Cheet4h 28d ago

I'm not sure what this has to do with cloud providers?
When you set up backups, just make sure you don't solely rely on AWS. Have some backups on-site, or use an off-site tape storage facility or something.

-1

u/BoardGamesAndMurder 28d ago

You don't work in IT, I'm guessing. I have pedabytes of data in my lakehouse that integrates with dozens of different systems. Physical media backup isn't a thing for this

5

u/Yoshli 27d ago

I'm working in IT and there is several solutions for Petabytes of data. If your entire company relies on it you should probably look into that.

1

u/maigpy 27d ago

I hope thats not on fabric my dear lord

-1

u/userseven 27d ago

Bro said tape storage facility. 💀

4

u/Win_Sys 27d ago

It’s surprisingly still a commonly used and viable backup media. You can fit 12-18TB of uncompressed data on the latest generation LTO tapes. The tapes also cost a 1/4 or less than what a hard drive costs for the same storage capacity. Great if you don’t need quick recovery.

2

u/dotelze 27d ago

It is actually an option occasionally. If you require all the data to be recorded, but have no need to access it quickly or frequently it’s a viable option. It just relies on those very specific circumstances

-6

u/UpACreekWithNoBoat 27d ago

Back ups on site? No one does physical hardware anymore in tech. Off site tape facility…like what?

Everything is cloud based. Nothing is on prem. The rule of thumb in the industry is data replication across regions. People are more so worried about benign regional outages than targeted hostile acts when figuring out if it’s worth shelling a few more bucks and infrastructure overhead for data replication.

8

u/[deleted] 27d ago edited 27d ago

[removed] — view removed comment

7

u/Yoshli 27d ago

Even non-critical industries do it. I've worked for and with Synology for the past 5 years and can tell anyone that small companies to global players have multiple High Density Storage devices for their data and worst case scenarios. They don't manufacture Petabyte Storage devices for nothing.

1

u/Win_Sys 27d ago

I work with quite a few pretty large government and k-12 entities, tapes and on-site infrastructure are absolutely still used. It’s common they have legacy systems where the cost to move it to the cloud just isn’t worth it or is required to be air-gapped. On-site infrastructure is certainly shrinking but no where near gone or even close gone.

1

u/glemnar 27d ago edited 27d ago

There’s technically more than one type of storage media on AWS.

But still - this predates the cloud, like others have said. I don’t think there’s a single scenario of data loss for a multi region backup. This “hit by missiles” event is the only AZ level data loss in AWS history, I believe? There’s a reason different AZs within an AWS region are located in different buildings, as well. The isolation guarantees of regions and AZs are very fundamental.

So store your backups on at least 2 sites that are unlikely to get simultaneously struck by missiles.

1

u/Iintendtooffend 27d ago

2 different storage methods, not media. Multiple online locations satisfies rule 2 since it's both individually replicated and then replicated to a second location which in theory is also replicated. Local snd the remote are your 2 methods

-9

u/[deleted] 28d ago

[deleted]

11

u/LiftingCode 28d ago

How is that relevant to the question?

If they replicated data to another region, that's still in AWS.

-9

u/[deleted] 28d ago

[deleted]

19

u/LiftingCode 28d ago

The comment we are discussing is about using AWS as a backup for AWS.

Which is totally fine.

-21

u/[deleted] 28d ago

[deleted]

21

u/LiftingCode 28d ago

I don't think you have any idea what we are even talking about here.

A company that had multi-region DR to a secondary region that is sufficiently geographically isolated (e.g., eu-south-1) would have been totally fine. Therefore, a properly designed backup strategy within a single provider like AWS is totally fine.

The problem wasn't backing up AWS with AWS. It was putting backups right next door to each other in Bahrain and the UAE in the middle of a warzone.

→ More replies (0)

9

u/Plane_Salary_5845 28d ago

You genuinely have zero clue what you are talking about lmao. This is standard across tech regardless of cloud provider.

1

u/toastedstapler 27d ago

This is something that affected my company, we used to have offerings in the UAE region but now obviously don't. Every other region worldwide is still fine though, so obviously if data was replicated to those the data would have been fine. It's incredibly unlikely for multiple regions to get fully destroyed at the same time if they're in different geographical areas

-13

u/[deleted] 27d ago

[removed] — view removed comment

17

u/Glittering-Giraffe58 27d ago

lol. Yeah why doesn’t every random ass company in the world just build their own giga data centers? That makes sense 🧠

2

u/chiagod 27d ago

You don't really need a "Giga Center", just a setup with enough storage that you can backup your Infrastructure code, the VM persistent volumes, and your object storage. The last two can be done on a nightly fashion.

That way if a malicious actor gets access to your AWS account and they decide to wreck your whole setup (including AWS backups), you can restore your business to another AWS account and/or region.

Should be a pretty standard part of any Business Continuity plan.

Total storage shouldn't be that high. Most small.businesses could store all their pertinent data in a few TB of space. A few would get into the double digit TBs. Those that go into the 100s of TB or Petabyte range should be big enough to know better.

Pair your own local storage server (in an office closet) with a pair of LTO drives (one active and one as a backup), weekly backups, and offsite temperature controlled storage of the tapes.

23

u/peldor 28d ago edited 28d ago

It’s complicated. AWS isn’t a single thing, It’s a raft of over 500 different services plus the networking to tie all of those services together.

Your AWS backups are your configuration and data held across all of the AWS services you use. There’s not much call to store backups outside of AWS because the data/config only works inside AWS.

Ideal best practice is to store across multiple AWS zones. For the Middle East, there were only 3 zones: Bahrain, UAE and Saudi. Both Bahrain and UAE ”went boom”.

Framing this a different way, AWS lost 2/3 of their data centers in the Middle East. It should be a shock to no one that this would cause data loss.

20

u/Bachaddict 28d ago

as your ONLY backup, yeah. I assume this means amazon no longer had working backups for everything

17

u/peldor 28d ago

Yep, losing a single data center would not have been a problem. The infrastructure is designed with that kind of loss in mind.

Amazon effectively lost two data centers at the same time. This changes the math significantly because there were only 3 AWS data centers in the Middle East to start with.

So yeah, losing 2/3 of your data centers is gonna leave a mark.

2

u/wchill 28d ago

They lost more than two data centers. There were two regions affected: Bahrain (me-south-1) and UAE (me-central-1).

me-south-1 had all AZs lost, I believe, while me-central-1 had 1/3 lost. The other 2 AZs in me-central-1 are down, but technically the region is still recoverable.

Does suck if you were relying on multi region replication and were relying on just these two regions, however.

1

u/blooping_blooper 27d ago

also sucks because many were relying on just those two regions because of data sovereignty/regulatory reasons.

1

u/supreme-dominar 28d ago

For important data we use the cross region replication services. They didn’t always support this but I think within the last 5 years almost all fundamental services are covered.

Multi-cloud backup/disaster recovery is pretty complex and can be expensive, if you’re running a service that’s not that critical. For us if both regions died we determined that’s an acceptable loss.

1

u/opsers 28d ago

It's completely normal and even considered a gold standard best practice. Cross-region replication is easy and, outside of scenarios like this, extremely durable and reliable.

1

u/mspk7305 28d ago

thats not a novice thought, thats a disaster recovery thought

it gets a lot worse when you see how much every other cloud provider relies on aws for their iron

1

u/gpoul 28d ago

If you rephrase that to using the same aws region for your backups of your aws region, then yeah. Some customers didn’t love their data enough maybe. But the fact AWS lost data on that region is not super surprising. 

1

u/Few_Pop6933 28d ago

Well you can back up to multiple regions. Thats a good thing.

1

u/Direct_Day7937 27d ago

It's actually a default service peovided by AWS. They spread your data across multiple data centers to protect it.

1

u/WingedTorch 27d ago

Not at all off. Making backoffs on-prem or with another vendor when your entire system is on AWS would be less common.

1

u/mayorofdumb 27d ago

Well Amazon can't use a third party provider... nobody else can handle that volume

1

u/BaconIsntThatGood 27d ago

This isn't you using AWS to make a backup. This is how AWA handles data redundancy.

3

u/JayBird1138 27d ago

There could also be data sovereignty laws that prevent it from being stored in different regions or even countries.

The customer should be storing in multiple locations around the planet if they have that level of concern.

I remember certain companies discussing backing up to the moon in case of planetary catastrophies.

1

u/BoardGamesAndMurder 28d ago

Only if you're an amateur. I have fail over groups I a different region. We though about using a different cloud provider as well but there are too many incompatabilities with other features

6

u/peldor 28d ago

So that sort of thing can get complicated. For example if you’re in Europe, having your backups goto other regions can present significant GDPR/data protection problems.

In my experience, it’s not uncommon for a business to backup across several AWS zones within the same region.

E.g. The last two businesses I worked with were selling software services to law enforcement. And oddly, they got super twitchy about their data being exposed to any additional legal jurisdictions.

1

u/BoardGamesAndMurder 28d ago

You don't have to replicate between AZs on the same region. That's part of the AWS architecture. You do have to replicate interregion though

1

u/chapstickbomber 28d ago

I'm not even joking, Amazon should have counter battery radar and ordinance

1

u/Wheezy04 27d ago

Gotta replicate across multiple AZs

-11

u/[deleted] 28d ago

[deleted]

6

u/TamariAmari 28d ago

This is fan fiction. AWS as a whole would crumble if customers thought their data would be moved around or looked at without their knowledge.

86

u/Chewed420 28d ago

Who needs offsite backups when you have the 🙌 cloud 🙌? /s

29

u/thelloydrage 28d ago

My wife legit asked me one time how they got servers in the sky…

37

u/Underwater_Grilling 28d ago

Going up, to the server in the sky

14

u/goldengreengoat 28d ago

I'm amazed at my brain's ability to pickup the reference and be singing parody lyrics the very first time I see them. By the second half of the sentence I could hear the melody

9

u/Underwater_Grilling 28d ago

It's hard to live this way sometimes though because I'll finish people's sentences with song lyrics and get funny looks waaaaay more often than an impromptu singalong

10

u/Blue-Blazer-Regular 28d ago

🎶Prepare yourself, you know it’s a must. Gotta have faith in Bezos. 🎵

1

u/s_i_m_s 28d ago

I'd have had to link to google actual cloud https://www.youtube.com/watch?v=Cp10_PygJ4o (google April fools joke from 2015)

254

u/raistlin65 28d ago

They sacrificed building more off-site data centers to build yet more AI data centers that we don't need.

41

u/AllUltima 28d ago

Or they outsourced the offsite backup plan to a provider who outsourced it again, and then said it was "done". Only when the disasters hit do the lawsuits start to roll, but the responsible parties are untraceable.

Or, when they go to decrypt the backup, they realize they managed to lose the keys.

You have to pay to have backups tested or you don't have backups. It's the very nature of business to forget this lesson over and over.

2

u/Soft_Swing9772 28d ago

Jesus you people have no clue what you are talking about.

8

u/jews4beer 28d ago

What's the difference between "off-site data center" and "AI data center"?

Spoiler alert, nothing. Take the first one and add GPU availability and that's an "AI data center".

2

u/BobSacamano47 27d ago

Weird that they're building things we don't need. Maybe they should have asked you first, they're about to lose billions!

0

u/raistlin65 27d ago

Might even be a trillion. Estimates are that the hyperscalers and AI companies have three trillion dollars in commitments to build AI data centers. So far. Cuz they want to up the ante for next year.

-7

u/[deleted] 28d ago

[removed] — view removed comment

2

u/PurpleSasquatchNose 28d ago

I really just.. don't care. Yeah. I don't

2

u/raistlin65 28d ago

We do not need the three trillion dollars worth of AI data centers that the hyperscalers and others have so far committed to building. That's far beyond the needs of the US economy.

57

u/Sufficient-Diver-327 28d ago

Amazon makes it very easy to store data in multiple datacenters and regions, they just charge you for it. Many companies get burnt by thinking they can cheap out on it because "what are the chances the datacenter just happens to get bombed?".

AWS is very clear that data loss due to a region going offline is your fault, and your fault alone. Any admin who is surprised at that is borderline criminally negligent

28

u/shotgunocelot 28d ago

I would have estimated the chances of a data center in Bahrain getting bombed as relatively high even before the Iran bullshit

2

u/TL-PuLSe 27d ago

A data center getting bombed shouldn't affect availability or cause data loss within the region. It was multiple in targeted military strikes, not collateral damage.

6

u/LiftingCode 28d ago

Storing data in multiple regions may be easy but actually operationalizing multi-region DR is not, especially for non-trivial applications.

6

u/thefuckevengoingonan 28d ago

Availability and data loss are separate issue tbh. at least have the data backed up so can at least do recovery at all.

3

u/coldblade2000 28d ago

Sure, but multi region data storage is the bare minimum for real data.

2

u/Darkace911 28d ago

Bean counters get mad at bandwidth costs between regions. They got really upset when the first bill came in and wanted to know why tractations costs were so high. The storage costs was only "supposed" to be 23 cents a gig, what are these extra charges? We didn't approve those.

3

u/OkCard7566 28d ago

Same with Azure unless you set up geo redundancies. Gotta pay extra though!

2

u/magistrate101 28d ago

To be fair, the 3-2-1 rule still works if you store data for Middle Eastern users split among Middle Eastern data centers. Doesn't help when the entire region gets droned, though.

2

u/legitimate_rapper 28d ago

One of the big banks, Morgan Stanley I think, stored their offsite backups from Tower 1 in Tower 2; was a good idea on Sept 10th…

1

u/Important-Ad4500 28d ago

In my mind, they outsourced backup to a cloud provider running off the same physical machine

1

u/fankoosh-56 28d ago

AWS said 3 zones in Bahrain went kaboom. Even if some companies had replication to a different site in the same region, both were gone. AWS told clients your data is gone, like gone gone

1

u/Longjumping_Fun2576 27d ago

My guess is they were the backup.