#331 - 17 April - Ongoing DDoS on code.f

#331 - 17 April - Ongoing DDoS on code.forgejo.org (updated 2 May) - forgejo/discussions - Codeberg.org
forgejo
discussions
Issues
190
Activity
17 April - Ongoing DDoS on code.forgejo.org (updated 2 May)
#331
New issue
Closed
opened
2025-04-17 07:41:10 +02:00
by
earl-warren
39 comments
earl-warren
commented
2025-04-17 07:41:10 +02:00
Copy link
Since 17 April, code.forgejo.org was disrupted by what appears to be a DDoS, similar to the
9 February event
. The service is not impacted yet.
There are ~35,000 different IPs per hour at the time of this message and a total of ~220,000 different IPs yesterday with peaks of ~15,000 different IPs per hour.
Since 17 April, code.forgejo.org was disrupted by what appears to be a DDoS, similar to the [9 February event](https://codeberg.org/forgejo/discussions/issues/297). The service is not impacted yet.

There are ~35,000 different IPs per hour at the time of this message and a total of ~220,000 different IPs yesterday with peaks of ~15,000 different IPs per hour.
rvba
commented
2025-04-17 12:20:35 +02:00
Copy link
Still
git blame
-ing ? Btw, found this kind of stuff when doing some research on this issue :
CarperAI is releasing a series of diff models—models trained to predict a code diff, trained on millions of commits scraped from GitHub.
Still ``git blame``-ing ? Btw, found this kind of stuff when doing some research on this issue : [CarperAI is releasing a series of diff models—models trained to predict a code diff, trained on millions of commits scraped from GitHub.](https://carper.ai/diff-models-a-new-way-to-edit-code/)
earl-warren
commented
2025-04-17 12:35:15 +02:00
Author
Copy link
It is all over the place, not just git blame.
It is all over the place, not just git blame.
earl-warren
commented
2025-04-17 12:45:43 +02:00
Author
Copy link
In the past 11 hours,
303,225 distinct IP
sent request to code.forgejo.org (between ~20,000 and ~40,000 per hour). The instance is running the latest Forgejo v11.0.0 release published yesterday (SQLite) and shows no sign of slowness yet. Here is the distribution of the top 30
IP ranges
contributing distinct IP (first column) to this DDoS. The distribution is roughly the same compared to yesterday and is clustered geographically, with the majority of ranges located in Vietnam and apparently used by telecom providers.
The top IP ranges will be blocked tomorrow if this keeps going on.
13495
7067
3397
1883
(datacenter)
1719
1558
1456
1423
1402
1315
786
679
628
611
610
605
594
493
491
475
466
441
435
415
399
377
371
356
346
321
In the past 11 hours, **303,225 distinct IP** sent request to code.forgejo.org (between ~20,000 and ~40,000 per hour). The instance is running the latest Forgejo v11.0.0 release published yesterday (SQLite) and shows no sign of slowness yet. Here is the distribution of the top 30 [IP ranges](https://code.forgejo.org/forgejo/ipranges#ipranges) contributing distinct IP (first column) to this DDoS. The distribution is roughly the same compared to yesterday and is clustered geographically, with the majority of ranges located in Vietnam and apparently used by telecom providers.

The top IP ranges will be blocked tomorrow if this keeps going on.

13495 https://bgpview.io/prefix/14.160.0.0/11
7067 https://bgpview.io/prefix/113.160.0.0/11
3397 https://bgpview.io/ip/123.16.0.0
1883 https://bgpview.io/prefix/38.0.0.0/8 (datacenter)
1719 https://bgpview.io/ip/14.240.0.0
1558 https://bgpview.io/ip/14.228.0.0
1456 https://bgpview.io/prefix/152.56.0.0/14
1423 https://bgpview.io/prefix/222.252.0.0/14
1402 https://bgpview.io/ip/14.232.0.0
1315 https://bgpview.io/ip/123.24.0.0
786 https://bgpview.io/prefix/213.230.64.0/18
679 https://bgpview.io/prefix/78.160.0.0/11
628 https://bgpview.io/prefix/171.224.0.0/11
611 https://bgpview.io/ip/14.226.0.0
610 https://bgpview.io/prefix/88.224.0.0/11
605 https://bgpview.io/prefix/37.236.0.0/14
594 https://bgpview.io/prefix/84.54.64.0/19
493 https://bgpview.io/prefix/196.188.0.0/14
491 https://bgpview.io/prefix/179.125.128.0/17
475 https://bgpview.io/prefix/169.224.0.0/17
466 https://bgpview.io/prefix/189.4.0.0/14
441 https://bgpview.io/prefix/177.37.128.0/17
435 https://bgpview.io/prefix/136.158.0.0/16
415 https://bgpview.io/prefix/73.0.0.0/8
399 https://bgpview.io/prefix/39.32.0.0/11
377 https://bgpview.io/prefix/187.19.128.0/17
371 https://bgpview.io/prefix/117.192.0.0/10
356 https://bgpview.io/ip/191.176.0.0
346 https://bgpview.io/prefix/49.32.0.0/13
321 https://bgpview.io/prefix/49.40.0.0/14
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 17 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 18 April)
2025-04-18 08:47:52 +02:00
earl-warren
commented
2025-04-18 08:52:43 +02:00
Author
Copy link
17 April the DDoS ~550,000 distinct IPs sent request to code.forgejo.org (between ~20,000 and ~40,000 per hour). It slowed down during the night, down under 10,000 per hour and then resumed with a steady ~30,000 distinct IP per hour.
Although code.forgejo.org sustains the load, the top IP ranges contributing to the DDoS will be blocked to relieve the pressure. They will be unblocked when the DDoS stops, hopefully in a week from now.
17 April the DDoS ~550,000 distinct IPs sent request to code.forgejo.org (between ~20,000 and ~40,000 per hour). It slowed down during the night, down under 10,000 per hour and then resumed with a steady ~30,000 distinct IP per hour.

Although code.forgejo.org sustains the load, the top IP ranges contributing to the DDoS will be blocked to relieve the pressure. They will be unblocked when the DDoS stops, hopefully in a week from now.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 18 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 19 April)
2025-04-19 09:18:55 +02:00
earl-warren
commented
2025-04-19 09:24:24 +02:00
Author
Copy link
18 April the DDoS ~400,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~35,000 per hour.
The top 30 IP ranges contributing to the DDoS have been blocked and reduce the pressure by 5% to 10%. During the previous DDoS between 300 and 500 IP ranges were blocked with an efficiency of 90%. There still is no need for that since code.forgejo.org keeps being responsive at all times.
18 April the DDoS ~400,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~35,000 per hour.

The top 30 IP ranges contributing to the DDoS have been blocked and reduce the pressure by 5% to 10%. During the previous DDoS between 300 and 500 IP ranges were blocked with an efficiency of 90%. There still is no need for that since code.forgejo.org keeps being responsive at all times.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 19 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 20 April)
2025-04-20 07:37:18 +02:00
earl-warren
commented
2025-04-20 08:12:35 +02:00
Author
Copy link
19 April the DDoS ~500,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~45,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked yesterday keep reducing the pressure by 5% to 10% which demonstrates that although IP are different, the IP ranges tend to be the same.
A working theory is that these DDoS originate from customers of
residential proxy service
providers. Since they are all priced by GB (in the range of 1USD to 4USD per GB), blocking them with a
403 error
is of no consequence to the customer: it is an empty response that does not cost them money. With an estimated average of 8KB per request and 2 requests per IP,
one day of DDoS is 8GB and estimated to cost between 8USD and 32USD
In an attempt to actively discourage the customers of those shady services, blocked IP ranges now receive an average of 20MB of random data instead of an empty 403 error. With 10% of IP blocked in this way, that would multiply the cost of the ongoing DDoS by ~1,000. They would download an estimated 3TB per day (0.02GB * 50,000 blocked IPs * 3 requests per IP), which costs between 3,000 USD and 12,000 USD per day.
A total of ~1.5TB was sent in the past 12 hours.
19 April the DDoS ~500,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~45,000 per hour.

The top 30 IP ranges contributing to the DDoS blocked yesterday keep reducing the pressure by 5% to 10% which demonstrates that although IP are different, the IP ranges tend to be the same.

A working theory is that these DDoS originate from customers of [residential proxy service](https://codeberg.org/forgejo/discussions/issues/320#issuecomment-3857070) providers. Since they are all priced by GB (in the range of 1USD to 4USD per GB), blocking them with a [403 error](https://developer.mozilla.org/de/docs/Web/HTTP/Reference/Status/403) is of no consequence to the customer: it is an empty response that does not cost them money. With an estimated average of 8KB per request and 2 requests per IP, **one day of DDoS is 8GB and estimated to cost between 8USD and 32USD**.

In an attempt to actively discourage the customers of those shady services, blocked IP ranges now receive an average of 20MB of random data instead of an empty 403 error. With 10% of IP blocked in this way, that would multiply the cost of the ongoing DDoS by ~1,000. They would download an estimated 3TB per day (0.02GB * 50,000 blocked IPs * 3 requests per IP), which costs between 3,000 USD and 12,000 USD per day.

A total of ~1.5TB was sent in the past 12 hours.

❤️
kita
commented
2025-04-20 10:00:33 +02:00
Member
Copy link
In an attempt to actively discourage the customers of those shady services, blocked IP ranges now receive an average of 20MB of random data instead of an empty 403 error
Wasting their time and even money at the same time is pure genius :3
> In an attempt to actively discourage the customers of those shady services, blocked IP ranges now receive an average of 20MB of random data instead of an empty 403 error

Wasting their time and even money at the same time is pure genius :3
earl-warren
commented
2025-04-21 09:22:25 +02:00
Author
Copy link
20 April the DDoS ~620,000 distinct IPs sent request to code.forgejo.org, in waves from ~10,000 per hour up to ~55,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 6% to 11% which demonstrates that although IP are different, the IP ranges tend to be the same (over three days).
The blocked IPs [have been sent random data](
#331 (comment)
) for a total of 2.2TB on 20 April, around 3.5TB in the past 48h. This exceeds the monthly GB allowance of all plans advertised by [residential proxy providers](
#320 (comment)
). There are two with 5TB plans advertised:
There is just one that
advertises plans up to 40TB
at a 0.3USD / GB rate.
And then there are a few with potential higher volume offers that requires contacting sales. According to the chatbot of one site, the proxy stops working when it runs out of credits.
Except if choosing the "pay as you go option" that is in the standard pack of all providers, but also the most costly (no less than 4USD per GB) and very unlikely to be the case.
20 April the DDoS ~620,000 distinct IPs sent request to code.forgejo.org, in waves from ~10,000 per hour up to ~55,000 per hour.

The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 6% to 11% which demonstrates that although IP are different, the IP ranges tend to be the same (over three days).

The blocked IPs [have been sent random data](https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3862101) for a total of 2.2TB on 20 April, around 3.5TB in the past 48h. This exceeds the monthly GB allowance of all plans advertised by [residential proxy providers](https://codeberg.org/forgejo/discussions/issues/320#issuecomment-3857070). There are two with 5TB plans advertised:

- https://dataimpulse.com/residential-proxies/#pricing
- https://rayobyte.com/products/residential-proxies#pricing

There is just one that [advertises plans up to 40TB](https://shifter.io/pricing) at a 0.3USD / GB rate.

![image](/attachments/dc8b5036-59c9-4433-8ca8-ccf35f6d6afe)

And then there are a few with potential higher volume offers that requires contacting sales. According to the chatbot of one site, the proxy stops working when it runs out of credits.

![image](/attachments/e2890b42-e2a8-45da-8776-821db86110ad)

Except if choosing the "pay as you go option" that is in the standard pack of all providers, but also the most costly (no less than 4USD per GB) and very unlikely to be the case.
image.png
44 KiB
image.png
47 KiB
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 20 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 21 April)
2025-04-21 11:24:36 +02:00
earl-warren
referenced this issue
2025-04-21 16:26:09 +02:00
Crawlers hitting Forgejo instances - global abuse trend
#320
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 21 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 22 April)
2025-04-22 09:44:57 +02:00
earl-warren
commented
2025-04-22 09:59:06 +02:00
Author
Copy link
21 April the DDoS ~320,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 11% which demonstrates that although IP are different, the IP ranges tend to be the same (over four days).
The blocked IPs
have been sent random data
for a total of ~5TB over the past 72h. This is the maximum of all publicly advertised plans and as the DDoS continues, it demonstrates that the author does not use one of these plans.
A 2024
summary
and
research paper
confirm that the residential proxy services which are likely to be used in this DDoS are priced by GB (~0.1USD/GB for the person providing the node and has to be sold more to make a profit).
21 April the DDoS ~320,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.

The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 11% which demonstrates that although IP are different, the IP ranges tend to be the same (over four days).

The blocked IPs [have been sent random data](https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3862101) for a total of ~5TB over the past 72h. This is the maximum of all publicly advertised plans and as the DDoS continues, it demonstrates that the author does not use one of these plans.

A 2024 [summary](https://www.orangecyberdefense.com/global/blog/research/residential-proxies) and [research paper](https://arxiv.org/pdf/2404.10610) confirm that the residential proxy services which are likely to be used in this DDoS are priced by GB (~0.1USD/GB for the person providing the node and has to be sold more to make a profit).
earl-warren
commented
2025-04-22 13:24:15 +02:00
Author
Copy link
The amount of data sent back to the blocked IP was incorrectly calculated and is actually 3 times lower. The accumulated volume is currently 2.03TB. The comments were updated accordingly. It brings estimations back to when the largest standard residential proxies plans were estimated to be exhausted. Here are the actual volumes sent per day, for a median size of random data of 5MB per request, over ~170,000 requests.
The amount of data sent back to the blocked IP was incorrectly calculated and is actually 3 times lower. The accumulated volume is currently 2.03TB. The comments were updated accordingly. It brings estimations back to when the largest standard residential proxies plans were estimated to be exhausted. Here are the actual volumes sent per day, for a median size of random data of 5MB per request, over ~170,000 requests.

![image](/attachments/416c34bf-32d5-42c0-a08a-a50303614efd)
image.png
37 KiB
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 22 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 23 April)
2025-04-23 09:06:59 +02:00
earl-warren
commented
2025-04-23 09:10:54 +02:00
Author
Copy link
22 April the DDoS ~370,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 11% which demonstrates that although IP are different, the IP ranges tend to be the same (over five days).
The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~2.5TB over the past four days.
22 April the DDoS ~370,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.

The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~2.5TB over the past four days.
earl-warren
referenced this issue
2025-04-23 17:46:47 +02:00
Crawlers hitting Forgejo instances - global abuse trend
#320
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 23 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 24 April)
2025-04-24 08:46:14 +02:00
earl-warren
commented
2025-04-24 08:50:15 +02:00
Author
Copy link
23 April the DDoS ~370,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 12% which demonstrates that although IP are different, the IP ranges tend to be the same (over six days).
The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~3TB over the past five days.
An IP range participating in the DDoS was redirected (302) to an observable destination in the past 24h. It received ~500 requests from the author of the DDoS and never followed the redirect.
23 April the DDoS ~370,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.

The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 12% which demonstrates that although IP are different, the IP ranges tend to be the same (over six days).

The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~3TB over the past five days.

An IP range participating in the DDoS was redirected (302) to an observable destination in the past 24h. It received ~500 requests from the author of the DDoS and never followed the redirect.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 24 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 25 April)
2025-04-25 08:12:47 +02:00
earl-warren
commented
2025-04-25 08:14:09 +02:00
Author
Copy link
24 April the DDoS ~330,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.
The top 30 IP ranges contributing to the DDoS blocked 18 April keep reducing the pressure by 5% to 12% which demonstrates that although IP are different, the IP ranges tend to be the same (over seven days).
The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~3.5TB over the past six days.
24 April the DDoS ~330,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour.

The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~3.5TB over the past six days.
bkil
commented
2025-04-25 09:07:36 +02:00
Copy link
Could you share the nature of the random data you respond with? Did you consider my advice above regarding poisoning of their database?
Could you share the nature of the random data you respond with? Did you consider my advice above regarding poisoning of their database?
earl-warren
commented
2025-04-25 10:26:31 +02:00
Author
Copy link
Could you share the nature of the random data you respond with?
dd if=/dev/urandom
Did you consider my advice above regarding poisoning of their database?
Yes. Would you like to help implement that?
> Could you share the nature of the random data you respond with?

`dd if=/dev/urandom`

> Did you consider my advice above regarding poisoning of their database?

Yes. Would you like to help implement that?
sclu1034
commented
2025-04-25 11:00:25 +02:00
Member
Copy link
At this point, I'm wondering what the goal is for the other end. You've labeled this as DDoS so far, and it did look like one in the beginning.
But it's been going on for a full week now, and the site is running just fine. They would have noticed by now that denial of service isn't going to happen.
So whatever they
are
up to seems to still be on track in their eyes, and worth the increase in price caused by your countermeasures. And if that's not DDoS, "they" might not even be one entity.
If this were several/many crawlers/scrapers that all just happened to put code.forgejo.org on their list at the same time (e.g. by building off of some public/shared dataset), this could also explain why the increase in bandwidth has no apparent effect: The larger the number of different customers, the more the increase in bandwidth will distribute over them and come less close to their quotas.
At this point, I'm wondering what the goal is for the other end. You've labeled this as DDoS so far, and it did look like one in the beginning.
But it's been going on for a full week now, and the site is running just fine. They would have noticed by now that denial of service isn't going to happen.

So whatever they _are_ up to seems to still be on track in their eyes, and worth the increase in price caused by your countermeasures. And if that's not DDoS, "they" might not even be one entity.

If this were several/many crawlers/scrapers that all just happened to put code.forgejo.org on their list at the same time (e.g. by building off of some public/shared dataset), this could also explain why the increase in bandwidth has no apparent effect: The larger the number of different customers, the more the increase in bandwidth will distribute over them and come less close to their quotas.
bkil
commented
2025-04-25 11:22:20 +02:00
Copy link
@earl-warren
I'd gladly experiment with sample and grammar-based filler content generation, but I've stopped doing go development in my free time.
@earl-warren I'd gladly experiment with sample and grammar-based filler content generation, but I've stopped doing go development in my free time.
bkil
commented
2025-04-25 11:27:21 +02:00
Copy link
@sclu1034
Some of the residential proxy providers lease bot nets, not volunteer computing resources, so their costs wouldn't increase.
As mentioned above, some proxy providers detect blocking such as via response code, mime type (or even magic number?) and not charge the customer for a response which is not valid and automatically retry each such request from another IP for them.
I still hold the hypothesis that Codeberg is collateral damage, not a target of an attack. If they wanted to tie up as much resources as possible, they would just send a request and immediately terminate the connection once the first response packet arrives. They would also schedule as many simultaneous connections from as many IPs as possible in a short time frame to cause an OOM or space exhaustion on the backend and sleep until the system is back up instead of spreading their load all day long.
@earl-warren
At the same time, as Codeberg already triggers Anubis, why block IPs specifically? It should already tie down their processing queue. And it's still an option to tie down the resources of a suspected abuser in multiple ways: such as by first including 100 tiny assets (to increase fetch count via img, iframe, CSS, fonts, json, etc.) and/or fetch them from JS, serving that 5MB HTML file to download and only then serve PoW at the end.
@sclu1034
Some of the residential proxy providers lease bot nets, not volunteer computing resources, so their costs wouldn't increase.

As mentioned above, some proxy providers detect blocking such as via response code, mime type (or even magic number?) and not charge the customer for a response which is not valid and automatically retry each such request from another IP for them.

I still hold the hypothesis that Codeberg is collateral damage, not a target of an attack. If they wanted to tie up as much resources as possible, they would just send a request and immediately terminate the connection once the first response packet arrives. They would also schedule as many simultaneous connections from as many IPs as possible in a short time frame to cause an OOM or space exhaustion on the backend and sleep until the system is back up instead of spreading their load all day long.

---

@earl-warren At the same time, as Codeberg already triggers Anubis, why block IPs specifically? It should already tie down their processing queue. And it's still an option to tie down the resources of a suspected abuser in multiple ways: such as by first including 100 tiny assets (to increase fetch count via img, iframe, CSS, fonts, json, etc.) and/or fetch them from JS, serving that 5MB HTML file to download and only then serve PoW at the end.
bziemons
commented
2025-04-25 11:34:11 +02:00
Copy link
@bkil
wrote in
#331 (comment)
@earl-warren
At the same time, as Codeberg already triggers Anubis, why block IPs specifically? It should already tie down their processing queue. And it's still an option to tie down the resources of a suspected abuser in multiple ways: such as by first including 100 tiny assets (to increase fetch count via img, iframe, CSS, fonts, json, etc.) and/or fetch them from JS, serving that 5MB HTML file to download and only then serve PoW at the end.
I don't want to disturb the conversation, but it was already addressed that this is not about Codeberg:
@earl-warren
wrote in
#320 (comment)
so making a step in the wrong direction by Codeberg would not be a great move.
There may be a confusion about the scope of my own comments. My first hand experience, the data I observe and the measure that were taken are exclusively focused on the infrastructure that is independent of Codeberg and 100% dedicated to the Forgejo project. I do not have access to the Codeberg infrastructure and have no knowledge to share.
@bkil wrote in https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3961850:

> @earl-warren At the same time, as Codeberg already triggers Anubis, why block IPs specifically? It should already tie down their processing queue. And it's still an option to tie down the resources of a suspected abuser in multiple ways: such as by first including 100 tiny assets (to increase fetch count via img, iframe, CSS, fonts, json, etc.) and/or fetch them from JS, serving that 5MB HTML file to download and only then serve PoW at the end.

I don't want to disturb the conversation, but it was already addressed that this is not about Codeberg:

@earl-warren wrote in https://codeberg.org/forgejo/discussions/issues/320#issuecomment-3934559:

> > so making a step in the wrong direction by Codeberg would not be a great move.
> There may be a confusion about the scope of my own comments. My first hand experience, the data I observe and the measure that were taken are exclusively focused on the infrastructure that is independent of Codeberg and 100% dedicated to the Forgejo project. I do not have access to the Codeberg infrastructure and have no knowledge to share.
earl-warren
commented
2025-04-25 12:28:31 +02:00
Author
Copy link
@sclu1034
wrote in
#331 (comment)
At this point, I'm wondering what the goal is for the other end. You've labeled this as DDoS so far, and it did look like one in the beginning. But it's been going on for a full week now, and the site is running just fine. They would have noticed by now that denial of service isn't going to happen.
This is a fair assessment. At this point it looks more like a crawler using residential proxies in a way that is so aggressive that it can be mistaken to be a DDoS. The only reason code.forgejo.org is holding is because measures were taken to improve the resilience since the previous event, two months ago. It intermittently brought down code.forgejo.org a few times a day, and only sent half the requests that are seen this week.
So whatever they
are
up to seems to still be on track in their eyes,
Yes.
and worth the increase in price caused by your countermeasures.
That is unclear as the total data sent still is under 5TB. It however demonstrates they are not using a standard plan as advertised on the net.
And if that's not DDoS, "they" might not even be one entity.
There is one hint that suggests it is a single entity using a single subscription: the blocked IP ranges have been operating at the same pace exactly for the past days. They are still the top 30 in activity, still block the same ration of the requests. If there were multiple actors I would not expect that to be so steady.
If this were several/many crawlers/scrapers that all just happened to put code.forgejo.org on their list at the same time (e.g. by building off of some public/shared dataset), this could also explain why the increase in bandwidth has no apparent effect: The larger the number of different customers, the more the increase in bandwidth will distribute over them and come less close to their quotas.
As the crawl-that-has-the-same-shape-as-a-DDoS continues, I'm tempted to increase the blocking of IP to raise the data cost. If it goes above 10TB, that will strongly suggest it has no significant impact on the cost for some reason.
@sclu1034 wrote in https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3961526:

> At this point, I'm wondering what the goal is for the other end. You've labeled this as DDoS so far, and it did look like one in the beginning. But it's been going on for a full week now, and the site is running just fine. They would have noticed by now that denial of service isn't going to happen.

This is a fair assessment. At this point it looks more like a crawler using residential proxies in a way that is so aggressive that it can be mistaken to be a DDoS. The only reason code.forgejo.org is holding is because measures were taken to improve the resilience since the previous event, two months ago. It intermittently brought down code.forgejo.org a few times a day, and only sent half the requests that are seen this week.

> So whatever they _are_ up to seems to still be on track in their eyes,

Yes.

> and worth the increase in price caused by your countermeasures.

That is unclear as the total data sent still is under 5TB. It however demonstrates they are not using a standard plan as advertised on the net.

> And if that's not DDoS, "they" might not even be one entity.

There is one hint that suggests it is a single entity using a single subscription: the blocked IP ranges have been operating at the same pace exactly for the past days. They are still the top 30 in activity, still block the same ration of the requests. If there were multiple actors I would not expect that to be so steady.

> If this were several/many crawlers/scrapers that all just happened to put code.forgejo.org on their list at the same time (e.g. by building off of some public/shared dataset), this could also explain why the increase in bandwidth has no apparent effect: The larger the number of different customers, the more the increase in bandwidth will distribute over them and come less close to their quotas.

As the crawl-that-has-the-same-shape-as-a-DDoS continues, I'm tempted to increase the blocking of IP to raise the data cost. If it goes above 10TB, that will strongly suggest it has no significant impact on the cost for some reason.
sclu1034
commented
2025-04-25 14:01:48 +02:00
Member
Copy link
@bkil
wrote in
#331 (comment)
Some of the residential proxy providers lease bot nets, not volunteer computing resources, so their costs wouldn't increase.
I wasn't talking about compute. earl-warren has been emphasizing that "the residential proxy services which are likely to be used in this DDoS are priced by GB", and sending large data blobs as a countermeasure is playing right into that.
proxy providers detect blocking [...] and not charge the customer [...] and automatically retry each such request from another IP for them
If that was happening, it should result in an overall increase in requests (due tue retries), and them being spread more evenly across the IPs available to the service (since every blocked request retries on an unblocked IP).
But we don't see that. As outlined in the daily posts, the number of unique IPs and their distribution across ranges have been quite even across the last few days. And while earl-warren hasn't shared detailed stats on request count, the amount of data sent in these requests has been the same each day, too.
And even if the proxy service does manage to detect and filter the random data responses, those still consumed bandwidth on their endpoints, which are also paid out by GB.
I still hold the hypothesis that Codeberg is collateral damage, not a target of an attack. [...] they would just send a request and immediately terminate the connection [...] schedule as many simultaneous connections from as many IPs as possible in a short time frame
As I was saying, the fact that it's steady and ongoing is why I don't believe it's a DDoS, and why I am wondering what the real goal might be.
But what would be a scenario where this is an attack at someone else, but also makes code.forgejo.org a collateral? If this was a DDoS at another site, with code.forgejo.org just accidentally on the same list, shouldn't we still see the patterns of short bursts, high volume?
I doubt that this is any kind of "attack" at all. They must be gaining value from the requests in the way they are happening right now, otherwise I doubt that they would have been going on for several days already.
And as I said, "they" might be multiple independent crawlers/scrapers/whatever, maybe even enough where each one individually might have been small enough to not ring any alarm bells.
@bkil wrote in https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3961850:

> Some of the residential proxy providers lease bot nets, not volunteer computing resources, so their costs wouldn't increase.

I wasn't talking about compute. earl-warren has been emphasizing that "the residential proxy services which are likely to be used in this DDoS are priced by GB", and sending large data blobs as a countermeasure is playing right into that.

> proxy providers detect blocking [...] and not charge the customer [...] and automatically retry each such request from another IP for them

If that was happening, it should result in an overall increase in requests (due tue retries), and them being spread more evenly across the IPs available to the service (since every blocked request retries on an unblocked IP).
But we don't see that. As outlined in the daily posts, the number of unique IPs and their distribution across ranges have been quite even across the last few days. And while earl-warren hasn't shared detailed stats on request count, the amount of data sent in these requests has been the same each day, too.

And even if the proxy service does manage to detect and filter the random data responses, those still consumed bandwidth on their endpoints, which are also paid out by GB.

> I still hold the hypothesis that Codeberg is collateral damage, not a target of an attack. [...] they would just send a request and immediately terminate the connection [...] schedule as many simultaneous connections from as many IPs as possible in a short time frame

As I was saying, the fact that it's steady and ongoing is why I don't believe it's a DDoS, and why I am wondering what the real goal might be.
But what would be a scenario where this is an attack at someone else, but also makes code.forgejo.org a collateral? If this was a DDoS at another site, with code.forgejo.org just accidentally on the same list, shouldn't we still see the patterns of short bursts, high volume?

I doubt that this is any kind of "attack" at all. They must be gaining value from the requests in the way they are happening right now, otherwise I doubt that they would have been going on for several days already.
And as I said, "they" might be multiple independent crawlers/scrapers/whatever, maybe even enough where each one individually might have been small enough to not ring any alarm bells.
sclu1034
commented
2025-04-25 14:16:12 +02:00
Member
Copy link
@earl-warren
wrote in
#331 (comment)
There is one hint that suggests it is a single entity using a single subscription: the blocked IP ranges have been operating at the same pace exactly for the past days. They are still the top 30 in activity, still block the same ration of the requests. If there were multiple actors I would not expect that to be so steady.
But only if the actors had a say in selecting the endpoints, right?
Those ranges might just be the ones with the most or best endpoints as determined by the proxy service, and thereby selected for all customers. Or it might even be an outside factor that makes those ranges popular across multiple proxy providers, e.g. demographics that makes the people behind those IPs more likely to buy into the business model.
And couldn't you also make the opposite argument? If this was a single actor, they would have the same complete view of requests that fail or succeed (e.g. based on response size, or termination before reaching the end of random data) as you do, would therefore be able to determine which ranges are blocked, and move off of them.
Whereas multiple actors, with requests spread randomly (potentially unevenly) would have a harder time noticing the patterns.
@earl-warren wrote in https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3962342:

> There is one hint that suggests it is a single entity using a single subscription: the blocked IP ranges have been operating at the same pace exactly for the past days. They are still the top 30 in activity, still block the same ration of the requests. If there were multiple actors I would not expect that to be so steady.

But only if the actors had a say in selecting the endpoints, right?
Those ranges might just be the ones with the most or best endpoints as determined by the proxy service, and thereby selected for all customers. Or it might even be an outside factor that makes those ranges popular across multiple proxy providers, e.g. demographics that makes the people behind those IPs more likely to buy into the business model.

And couldn't you also make the opposite argument? If this was a single actor, they would have the same complete view of requests that fail or succeed (e.g. based on response size, or termination before reaching the end of random data) as you do, would therefore be able to determine which ranges are blocked, and move off of them.
Whereas multiple actors, with requests spread randomly (potentially unevenly) would have a harder time noticing the patterns.
earl-warren
commented
2025-04-25 15:19:58 +02:00
Author
Copy link
@sclu1034
wrote in
#331 (comment)
All valid points. It is very difficult to draw definitive conclusions at this point.
@sclu1034 wrote in https://codeberg.org/forgejo/discussions/issues/331#issuecomment-3964655:

All valid points. It is very difficult to draw definitive conclusions at this point.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 25 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 26 April)
2025-04-26 07:28:00 +02:00
earl-warren
commented
2025-04-26 07:39:51 +02:00
Author
Copy link
25 April the DDoS showed signed of continuing at the same pace (~300,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour).
An experiment of three days was started towards the end of the day to block ~40% of the IPs rather than ~10% in order to increase the amount of data sent to the author of the DDoS. It will end 29 April. A change of pattern was observed, with a 50% increase in the number of IPs which suggests they are rotating faster as a result of being blocked.
The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~5.5TB over the past seven days.
25 April the DDoS showed signed of continuing at the same pace (~300,000 distinct IPs sent request to code.forgejo.org, in waves from ~5,000 per hour up to ~30,000 per hour).

An experiment of three days was started towards the end of the day to block ~40% of the IPs rather than ~10% in order to increase the amount of data sent to the author of the DDoS. It will end 29 April. A change of pattern was observed, with a 50% increase in the number of IPs which suggests they are rotating faster as a result of being blocked.

The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~5.5TB over the past seven days.
sclu1034
commented
2025-04-26 14:33:35 +02:00
Member
Copy link
How did the distribution of IPs across subnets change after the increase? Are they distributed similarly to before, or did they actively rotate toward unblocked subnets, thereby spreading the distribution more evenly?
And was there an increase in parallel requests, or do they appear to be sequential retries?
Some theories to pit against each other are
they figured out that certain subnets were blocked and actively moved away from them
the random data triggers a timeout or payload threshold, which in turn triggers a retry, which in turn means more requests total that the provider just happens to put on more IPs
the actor(s) send requests at an interval that is shorter than it takes for the random data responses to finish, therefore more requests are running in parallel, which require more IPs overall
And I guess another question in that regard: Does the number of requests per day correlate to the number of unique IPs per day?
How did the distribution of IPs across subnets change after the increase? Are they distributed similarly to before, or did they actively rotate toward unblocked subnets, thereby spreading the distribution more evenly?
And was there an increase in parallel requests, or do they appear to be sequential retries?

Some theories to pit against each other are
- they figured out that certain subnets were blocked and actively moved away from them
- the random data triggers a timeout or payload threshold, which in turn triggers a retry, which in turn means more requests total that the provider just happens to put on more IPs
- the actor(s) send requests at an interval that is shorter than it takes for the random data responses to finish, therefore more requests are running in parallel, which require more IPs overall

And I guess another question in that regard: Does the number of requests per day correlate to the number of unique IPs per day?
earl-warren
commented
2025-04-26 22:29:21 +02:00
Author
Copy link
How did the distribution of IPs across subnets change after the increase? Are they distributed similarly to before, or did they actively rotate toward unblocked subnets, thereby spreading the distribution more evenly?
The table below represents unique IPs.
date/UTC
accepted
blocked
total
blocked%
26/Apr/2025:
297468
179336
476804
37%
26/Apr/2025:00:
25973
17239
43212
39%
26/Apr/2025:01:
25448
16378
41826
39%
26/Apr/2025:02:
25660
16396
42056
38%
26/Apr/2025:03:
25687
16286
41973
38%
26/Apr/2025:04:
16872
11123
27995
39%
26/Apr/2025:05:
6052
4616
10668
43%
26/Apr/2025:06:
6284
4495
10779
41%
26/Apr/2025:07:
5687
4438
10125
43%
26/Apr/2025:08:
6175
4657
10832
42%
26/Apr/2025:09:
22776
15299
38075
40%
26/Apr/2025:10:
25830
17659
43489
40%
26/Apr/2025:11:
25914
17451
43365
40%
26/Apr/2025:12:
25519
18085
43604
41%
26/Apr/2025:13:
22640
15550
38190
40%
26/Apr/2025:14:
6314
4926
11240
43%
26/Apr/2025:15:
6385
4822
11207
43%
26/Apr/2025:16:
6238
5161
11399
45%
26/Apr/2025:17:
5691
5000
10691
46%
26/Apr/2025:18:
13122
10146
23268
43%
26/Apr/2025:19:
26237
19161
45398
42%
26/Apr/2025:20:
10363
7711
18074
42%
this is how it looks today. So, overall, the subnets are blocked in the same proportion today as they were yesterday. They did not significantly rotate, although some of them did. But a fraction that does not really make a difference.
And was there an increase in parallel requests, or do they appear to be sequential retries?
There still is an average of 4 request per IP on any given day, sequential for the most part.
the random data triggers a timeout or payload threshold, which in turn triggers a retry, which in turn means more requests total that the provider just happens to put on more IPs
That theory has my preference. But it is really difficult to conclude at this point.
And I guess another question in that regard: Does the number of requests per day correlate to the number of unique IPs per day?
Yes. The vast majority of requests daily are in direct proportion of the number of IPs. When there is no crawl-DDoS the number of requests and IP are orders of magnitude smaller (thousands rather than millions).
> How did the distribution of IPs across subnets change after the increase? Are they distributed similarly to before, or did they actively rotate toward unblocked subnets, thereby spreading the distribution more evenly?

The table below represents unique IPs.

| date/UTC | accepted | blocked | total | blocked% |
|---|---|---|---|---|
| 26/Apr/2025: | 297468 | 179336 | 476804 | 37% |
| 26/Apr/2025:00: | 25973 | 17239 | 43212 | 39% |
| 26/Apr/2025:01: | 25448 | 16378 | 41826 | 39% |
| 26/Apr/2025:02: | 25660 | 16396 | 42056 | 38% |
| 26/Apr/2025:03: | 25687 | 16286 | 41973 | 38% |
| 26/Apr/2025:04: | 16872 | 11123 | 27995 | 39% |
| 26/Apr/2025:05: | 6052 | 4616 | 10668 | 43% |
| 26/Apr/2025:06: | 6284 | 4495 | 10779 | 41% |
| 26/Apr/2025:07: | 5687 | 4438 | 10125 | 43% |
| 26/Apr/2025:08: | 6175 | 4657 | 10832 | 42% |
| 26/Apr/2025:09: | 22776 | 15299 | 38075 | 40% |
| 26/Apr/2025:10: | 25830 | 17659 | 43489 | 40% |
| 26/Apr/2025:11: | 25914 | 17451 | 43365 | 40% |
| 26/Apr/2025:12: | 25519 | 18085 | 43604 | 41% |
| 26/Apr/2025:13: | 22640 | 15550 | 38190 | 40% |
| 26/Apr/2025:14: | 6314 | 4926 | 11240 | 43% |
| 26/Apr/2025:15: | 6385 | 4822 | 11207 | 43% |
| 26/Apr/2025:16: | 6238 | 5161 | 11399 | 45% |
| 26/Apr/2025:17: | 5691 | 5000 | 10691 | 46% |
| 26/Apr/2025:18: | 13122 | 10146 | 23268 | 43% |
| 26/Apr/2025:19: | 26237 | 19161 | 45398 | 42% |
| 26/Apr/2025:20: | 10363 | 7711 | 18074 | 42% |

this is how it looks today. So, overall, the subnets are blocked in the same proportion today as they were yesterday. They did not significantly rotate, although some of them did. But a fraction that does not really make a difference.

> And was there an increase in parallel requests, or do they appear to be sequential retries?

There still is an average of 4 request per IP on any given day, sequential for the most part.

> the random data triggers a timeout or payload threshold, which in turn triggers a retry, which in turn means more requests total that the provider just happens to put on more IPs

That theory has my preference. But it is really difficult to conclude at this point.

> And I guess another question in that regard: Does the number of requests per day correlate to the number of unique IPs per day?

Yes. The vast majority of requests daily are in direct proportion of the number of IPs. When there is no crawl-DDoS the number of requests and IP are orders of magnitude smaller (thousands rather than millions).
earl-warren
commented
2025-04-27 09:56:42 +02:00
Author
Copy link
26 April the DDoS showed signed of increasing (~550,000 distinct IPs sent request to code.forgejo.org, in waves from ~10,000 per hour up to ~45,000 per hour).
The experiment started yesterday to block ~40% of the IPs rather than ~10% in order to increase the amount of data sent to the author of the DDoS was stopped after sending 8TB in 24h. The blocked IP are down to ~10%.
The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~11TB over the past eight days.
26 April the DDoS showed signed of increasing (~550,000 distinct IPs sent request to code.forgejo.org, in waves from ~10,000 per hour up to ~45,000 per hour).

The experiment started yesterday to block ~40% of the IPs rather than ~10% in order to increase the amount of data sent to the author of the DDoS was stopped after sending 8TB in 24h. The blocked IP are down to ~10%.

The blocked IP are sent a median size of random data of 5MB per request since 19 April and totals ~11TB over the past eight days.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 26 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 27 April)
2025-04-27 10:13:59 +02:00
earl-warren
referenced this issue
2025-04-27 10:26:57 +02:00
Crawlers hitting Forgejo instances - global abuse trend
#320
earl-warren
referenced this issue from forgejo/website
2025-04-27 11:04:39 +02:00
Monthly Update for April 2025
#571
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 27 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 28 April)
2025-04-28 08:27:04 +02:00
earl-warren
commented
2025-04-28 08:31:18 +02:00
Author
Copy link
27 April the DDoS is still very active (~550,000 distinct IPs sent request to code.forgejo.org). It started with with around 7,000 IPs per hour until 4am and the rest of the day was a steady 30,000 IPs per hour without interruptions.
The blocked IP are between ~8% and ~13. They are sent a median size of random data of 5MB per request for a total of 1TB. The accumulated volume since 19 April is ~12TB.
27 April the DDoS is still very active (~550,000 distinct IPs sent request to code.forgejo.org). It started with with around 7,000 IPs per hour until 4am and the rest of the day was a steady 30,000 IPs per hour without interruptions.

The blocked IP are between ~8% and ~13. They are sent a median size of random data of 5MB per request for a total of 1TB. The accumulated volume since 19 April is ~12TB.
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 28 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 29 April)
2025-04-29 09:12:07 +02:00
earl-warren
commented
2025-04-29 09:16:21 +02:00
Author
Copy link
28 April the DDoS slowed down (~200,000 distinct IPs sent request to code.forgejo.org). It started with with around 7,000 IPs per hour until 8pm and the rest of the day rose again with a steady 24,000 IPs per hour.
The blocked IP blocked around ~8% until 4pm UTC and went up to blocking between ~10% and 16% after that. They are sent a median size of random data of 5MB per request for a total of 0.4TB. The accumulated volume since 19 April is ~12.5TB.
28 April the DDoS slowed down (~200,000 distinct IPs sent request to code.forgejo.org). It started with with around 7,000 IPs per hour until 8pm and the rest of the day rose again with a steady 24,000 IPs per hour.

The blocked IP blocked around ~8% until 4pm UTC and went up to blocking between ~10% and 16% after that. They are sent a median size of random data of 5MB per request for a total of 0.4TB. The accumulated volume since 19 April is ~12.5TB.
litchipi
commented
2025-04-29 17:37:09 +02:00
Member
Copy link
Little comment just to thank you
@earl-warren
for handling this
❤️
It doesn't seam as complicated to handle as the previous DDoS, but still. Regarding counter-measures, once I get a bit more time I'll gladly help with the war effort.
Little comment just to thank you @earl-warren for handling this ❤️ It doesn't seam as complicated to handle as the previous DDoS, but still. Regarding counter-measures, once I get a bit more time I'll gladly help with the war effort.
❤️
earl-warren
commented
2025-04-30 07:00:55 +02:00
Author
Copy link
29 April the DDoS went up again (~330,000 distinct IPs sent request to code.forgejo.org). In waves between 6,000 and 25,000 unique IPs per hour.
The ratio of blocked IPs went slightly up and ranges from 9% to 25%. There are ~50 IP ranges blocked since 27 April and they were not modified. The fact that the percentage of blocked IP raises means that the DDoS started using more IPs in these ranges.
The blocked IP were sent a median size of random data of 5MB per request for a total of 0.8TB. The accumulated volume since 19 April is ~13.5TB.
29 April the DDoS went up again (~330,000 distinct IPs sent request to code.forgejo.org). In waves between 6,000 and 25,000 unique IPs per hour.

The ratio of blocked IPs went slightly up and ranges from 9% to 25%. There are ~50 IP ranges blocked since 27 April and they were not modified. The fact that the percentage of blocked IP raises means that the DDoS started using more IPs in these ranges.

The blocked IP were sent a median size of random data of 5MB per request for a total of 0.8TB. The accumulated volume since 19 April is ~13.5TB.
earl-warren
referenced this issue
2025-04-30 15:11:28 +02:00
Effective countermeasure against excessive crawling of Forgejo instances
#339
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 29 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 30 April)
2025-05-01 10:06:54 +02:00
earl-warren
commented
2025-05-01 10:13:36 +02:00
Author
Copy link
30 April the DDoS is still here (~200,000 distinct IPs sent request to code.forgejo.org). In the afternoon Anubis was setup and it significantly impacted its behavior. Within less than two hours the number of IPs went under 2,000 per hour while the percentage of IP originating from IP ranges that are blocked went up to 77%. This suggests that the crawler detects the presence of Anubis and stops altogether but does not conclude that random data sent is cause for giving up.
The blocked IP were sent a median size of random data of 5MB per request for a total of 0.5TB. The accumulated volume since 19 April is ~14TB.
30 April the DDoS is still here (~200,000 distinct IPs sent request to code.forgejo.org). In the afternoon Anubis was setup and it significantly impacted its behavior. Within less than two hours the number of IPs went under 2,000 per hour while the percentage of IP originating from IP ranges that are blocked went up to 77%. This suggests that the crawler detects the presence of Anubis and stops altogether but does not conclude that random data sent is cause for giving up.

The blocked IP were sent a median size of random data of 5MB per request for a total of 0.5TB. The accumulated volume since 19 April is ~14TB.
Snoweuph
commented
2025-05-01 13:36:55 +02:00
Copy link
how about also throwing some tarpits in the mix and also measure their performance in this DDOS?
how about also throwing some tarpits in the mix and also measure their performance in this DDOS?
earl-warren
commented
2025-05-01 16:37:44 +02:00
Author
Copy link
could do that. I'm not sure about the added value though. It would poison the collected data but will it poison it more than sending 5MB of random data on each request? It would also increase the number of URLs but there already are zillions of URLs on code.forgejo.org, more than enough apparently to keep the crawler busy during two weeks. Would it cost the crawler more than sending random data on existing URL? Maybe if the crawler runs out of URLs to request which it apparently does not. Would it shield code.forgejo.org more efficiently than Anubis? Probably not seeing that Anubis being activated seems to have scared away the crawler very rapidly.
In a nustshell it is unclear if it is worth the time. But if you are thinking of something that's more compelling, do tell!

In a nustshell it is unclear if it is worth the time. But if you are thinking of something that's more compelling, do tell!
Snoweuph
commented
2025-05-01 17:37:04 +02:00
Copy link
It was just an Idea, after the moto: lets try all the options
It was just an Idea, after the moto: lets try all the options
earl-warren
commented
2025-05-01 17:52:29 +02:00
Author
Copy link
Trying all the options requires an infinite amount of time.
Trying all the options requires an infinite amount of time.
Snoweuph
commented
2025-05-01 18:28:23 +02:00
Copy link
true true
true true
Snoweuph
commented
2025-05-01 18:28:24 +02:00
Copy link
true true
true true
earl-warren
changed title from
17 April - Ongoing DDoS on code.forgejo.org (updated 30 April)
to
17 April - Ongoing DDoS on code.forgejo.org (updated 2 May)
2025-05-02 16:06:26 +02:00
earl-warren
commented
2025-05-02 16:12:26 +02:00
Author
Copy link
1 May there still are traces of the DDoS (~3,000 IPs blocked in the known IP ranges it has used in the past days) but it is no longer disruptive with a total of ~10,000 unique IPs during the day. Its activity further decreased today with 9% of blocked IPs and no more than 1,500 unique IPs within the hour.
The blocked IP were sent a median size of random data of 5MB for an accumulated volume since 19 April is ~14TB.
A summary was drafted as a conclusion at
#339
1 May there still are traces of the DDoS (~3,000 IPs blocked in the known IP ranges it has used in the past days) but it is no longer disruptive with a total of ~10,000 unique IPs during the day. Its activity further decreased today with 9% of blocked IPs and no more than 1,500 unique IPs within the hour.

The blocked IP were sent a median size of random data of 5MB for an accumulated volume since 19 April is ~14TB.

A summary was drafted as a conclusion at https://codeberg.org/forgejo/discussions/issues/339
earl-warren
closed this issue
2025-05-02 16:12:28 +02:00
forgejo-actions
referenced this issue from forgejo/website
2025-11-27 18:11:38 +01:00
Dead links report
#529
to join this conversation.
No Branch/Tag specified
Branches
Tags
No results found.
No results found.
Labels
Clear labels
User research - Accessibility
Requires input about accessibility features, likely involves user testing.
User research - Blocked
Do not pick as-is! We are happy if you can help, but please coordinate with ongoing redesign in this area.
User research - Community
Community features, such as discovering other people's work or otherwise feeling welcome on a Forgejo instance.
User research - Config (instance)
Instance-wide configuration, authentication and other admin-only needs.
User research - Errors
How to deal with errors in the application and write helpful error messages.
User research - Filters
How filter and search is being worked with.
User research - Future backlog
The issue might be inspiring for future design work.
User research - Git workflow
AGit, fork-based and new Git workflow, PR creation etc
User research - Labels
Active research about Labels
User research - Moderation
Moderation Featuers for Admins are undergoing active User Research
User research - Needs input
Use this label to let the User Research team know their input is requested.
User research - Notifications/Dashboard
Research on how users should know what to do next.
User research - Rendering
Text rendering, markup languages etc
User research - Repo creation
Active research about the New Repo dialog.
User research - Repo units
The repo sections, disabling them and the "Add more" button.
User research - Security
User research - Settings (in-app)
How to structure in-app settings in the future?
No labels
User research - Accessibility
User research - Blocked
User research - Community
User research - Config (instance)
User research - Errors
User research - Filters
User research - Future backlog
User research - Git workflow
User research - Labels
User research - Moderation
User research - Needs input
User research - Notifications/Dashboard
User research - Rendering
User research - Repo creation
User research - Repo units
User research - Security
User research - Settings (in-app)
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
8 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Reference
forgejo/discussions#331
Reference in a new issue
No description provided.
Delete branch "%!s(
)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?