Preventing denial of service from screen scraping bots

[James_G]

Hi,

I manage a public service website for local government providing information on planning applications. Currently have an issue where every day we receive 8,000 hits from an IP address sourced in Amazon EC2 in a 30 minute window, each downloading possibly up to couple of MB in documents. Our system obviously struggles with this sudden request for up to 5GB data in a very small time window, causing system availability issues every day. We have reported the abuse to Amazon, but we don't expect results.

The source IP changes every day, and we can't block the whole of Amazon EC2 sources (I checked and Amazon publish over 800 subnets, and some of the events have been from sources outside the published list).

I have a DoS policy on the external interface, but this is not triggering as I never have a high level of concurrent sessions, the source is closing the sessions as it goes along. Is there a way to configure DoS policy to prevent number of sessions from a source per minute?

With reference to the above, anyone with recommendations on how to prevent this??

For reference my current DoS policy is:

config firewall DoS-policy edit 1 set interface "port3" set srcaddr "all" set dstaddr "all" set service "ALL" config anomaly edit "tcp_syn_flood" set status enable set log enable set action block set threshold 2000 next edit "tcp_port_scan" set status enable set log enable set action block set threshold 1000 next edit "tcp_src_session" set status enable set log enable set action block set threshold 5000 next edit "tcp_dst_session" set status enable set log enable set action block set threshold 5000 next edit "udp_flood" set status enable set log enable set action block set threshold 2000 next edit "udp_scan" set status enable set log enable set action block set threshold 2000 next edit "udp_src_session" set status enable set log enable set action block set threshold 5000 next edit "udp_dst_session" set status enable set log enable set action block set threshold 5000 next edit "icmp_flood" set status enable set log enable set action block set threshold 250 next edit "icmp_sweep" set status enable set log enable set action block set threshold 100 next edit "icmp_src_session" set status enable set log enable set action block set threshold 300 next edit "icmp_dst_session" set status enable set log enable set action block set threshold 1000 next edit "ip_src_session" set status enable set log enable set action block set threshold 5000 next edit "ip_dst_session" set status enable set log enable set action block set threshold 5000 next edit "sctp_flood" set status enable set log enable set action block set threshold 2000 next edit "sctp_scan" set status enable set log enable set action block set threshold 1000 next edit "sctp_src_session" set status enable set log enable set action block set threshold 5000 next edit "sctp_dst_session" set status enable set log enable set action block set threshold 5000 next end next end

[mahesh_secure]

Hi

which OS is running in firewall. on latest os there is an option internet service database and that have amazone ip address. you can create policy with this database and in that policy you can add traffic shape and session limit.

Regards

Mahesh

[James_G]

Right - I get it

I am on 5.4.4 at the moment, looks like an upgrade to 5.6 is incoming

[James_G]

I have upgraded - but only find the internet service in destination, not source

Any way I can limit the source addresses to the internet service list????