Romxnus753AC
Luminary
- Joined
- Nov 25, 2024
- Posts
- 5,108
- Reputation
- 4,315
i ran a controlled scraping test on my own account to see how the forum holds up against automated data collection. Wanted to share the results because they're not great
What we did: used a real browser session (my own, logged in), attached an automation tool to it, and let it go through the search results like a normal user browsing
(the script logged itself without human help which cloudflare should have detected )
First attempt got hit by the Cloudflare captcha, which is expected
Second attempt went through without any captcha at all. The browser session was already "trusted" by Cloudflare, so the script just kept going.
It pulled the full search history of one test account (mine) every thread, post, timestamp, forum ~900 records in a few minutes. No rate limit no block or resistence
The point isn't that we scraped our own data. The point is that if i can do it, anyone can do it on everybody here with one script(which i programmed) The forum currently doesn't stop automated traffic once a session is trusted, and that's a problem for everyone's data, so @Master and other mods need to update this forum, end points could be exposed, if i probabily dig i will find them, just update the forum or WE will lose all our data to some random hacker
so do this=
Update XenForo to the latest version (there are known CVEs fixed in 2.3.13+ that we should be on)
Configure Cloudflare WAF properly rate limiting bot management, not just the initial challenge
Add server side detection for abnormal browsing patterns (pageperfew-seconds, no mouse events, etc.)
Review what user data is actually exposed in search results and whether it needs to be
Not trying to fearmonger Just saying: we tested it, it worked way too easily, and we should fix it before someone with worse intentions does the same thing.
pictures:
for example this is my recent data scrapped, everything i posted is in the database, pictures, links, addresses, (ofc my own)
(if you ban me again, ur just exposing ur users to threats and its illigal)
What we did: used a real browser session (my own, logged in), attached an automation tool to it, and let it go through the search results like a normal user browsing
(the script logged itself without human help which cloudflare should have detected )
First attempt got hit by the Cloudflare captcha, which is expected
Second attempt went through without any captcha at all. The browser session was already "trusted" by Cloudflare, so the script just kept going.
It pulled the full search history of one test account (mine) every thread, post, timestamp, forum ~900 records in a few minutes. No rate limit no block or resistence
The point isn't that we scraped our own data. The point is that if i can do it, anyone can do it on everybody here with one script(which i programmed) The forum currently doesn't stop automated traffic once a session is trusted, and that's a problem for everyone's data, so @Master and other mods need to update this forum, end points could be exposed, if i probabily dig i will find them, just update the forum or WE will lose all our data to some random hacker
so do this=
Update XenForo to the latest version (there are known CVEs fixed in 2.3.13+ that we should be on)
Configure Cloudflare WAF properly rate limiting bot management, not just the initial challenge
Add server side detection for abnormal browsing patterns (pageperfew-seconds, no mouse events, etc.)
Review what user data is actually exposed in search results and whether it needs to be
Not trying to fearmonger Just saying: we tested it, it worked way too easily, and we should fix it before someone with worse intentions does the same thing.
pictures:
for example this is my recent data scrapped, everything i posted is in the database, pictures, links, addresses, (ofc my own)
(if you ban me again, ur just exposing ur users to threats and its illigal)
Last edited: