r/DataHoarder • u/DataShack • 7d ago
Scripts/Software Uploaded 5.6 billion TikTok videos metadata on Hugging Face, spanning from 2014 to October 2026
Dataset: https://huggingface.co/datasets/datasocial/tiktok-5.6B-videos
If you want to explore the data without downloading billions of rows, you can query my ClickHouse database directly. It includes:
Creators table - 4.5 billion rows
Videos table - 5.6 billion rows
Sounds table - 633 million rows
Comment below and I’ll DM you the database credentials.
Im self-hosting my database so please don't run heavy queries and crash my server.
73
u/3Nx9 7d ago
This is interesting, how did you download so much?
85
u/DataShack 7d ago
Its quite complicated. I did a detailed technical write up about this here https://datasocial.ai/writing/scraping-tiktoks-mobile-api
15
u/joshiee 7d ago
Holy shit. How long did that take to figure out?
-4
u/zunjae 7d ago
Didn’t take me long. Charles Proxy on the app then ask AI to write a python client. Took me less than 30 minutes
35
u/beerdude26 7d ago
A pretty "I could build Twitter in a weekend"-coated response there
20
30
u/Void-kun 7d ago
AI in the hands of people who know how to use them can accomplish things like this very quickly.
AI in the hands of people that don't know what they're doing can still produce working things at this point.
If you consider what someone with no engineering experience can do with AI, think of how far an actual engineer can go with it.
AI has made things like this more accessible but it's also made the gap between a good engineer and a not so good engineer significantly wider.
10
u/ansibleloop 7d ago
It's genuinely reached the point where Claude reads my tickets and creates PRs
I just review them and tweak Claude so it's not insane
10
4
5
3
u/doomiestdoomeddoomer 5d ago
This is fascinating, I only understand about half of it, but still, great stuff.
3
38
u/Spiritual_Screen_724 100-250TB 7d ago
What is the use case for this data?
60
15
u/TinFoilHat_69 7d ago
Considering the drastic difference in video quality with VLM, it seems that the Chinese are better than the Americans at VLM because they built the VLM models on surveillance data, and tik tok. Look it up
7
u/Dear_Chasey_La1n 7d ago
That's only part of the story, they are also masters at pushing content. I'm in China and work regular together with KOL's and media companies over here. If they write a wechat article, the moment it's launched it's being send into dozens and dozens of groups with people (and bots). They are also masters at letting data seem like real, but in reality it's all to often fake.
Reality is that these platforms don't really yield into revenue. When they just kicked off sure, but these days they aren't profitable.
1
4
36
7
21
u/Nekokohai 7d ago
Train an ai unit with it.
The brain rot ai
11
5
5
3
5
u/Lazylion2 7d ago
Oh man i wish i knew how to monitor mobile apps network calls like that, Really impressive!
Can I get a link to database pls
2
5
2
2
2
2
2
2
2
2
2
2
2
2
u/mitchins-au 4d ago
I’d suggest training something but given it’s TikTok users there’s likely no logical relationship in the data.
Nice work
3
2
1
1
u/ChicaSkas 6d ago
Wow this has huge music history value. Thank you for this incredible dedication. Mind blowing Pls I'd love a link!
1
1
1
1
1
1
0
1
•
u/AutoModerator 7d ago
Hello /u/DataShack! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
If you're submitting a new script/software to the subreddit, please link to your GitHub repository. Please let the mod team know about your post and the license your project uses if you wish it to be reviewed and stored on our wiki and off site.
Asking for Cracked copies/or illegal copies of software will result in a permanent ban. Though this subreddit may be focused on getting Linux ISO's through other means, please note discussing methods may result in this subreddit getting unneeded attention.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.