A proper DOS- Protection for Citrix Gateway

A

One of the main concerns that my large customers have is that the Citrix Gateway could fall victim to a DOS or DDOS attack. Linked to this, of course, is the concern that – after a successful attack – it might be possible to bypass authentication or compromise the gateway or the appliance.

We have to distinguish between attacks that happen before and those that happen after authentication. Risk assessment is not difficult here: we trust the authenticated users, so we fear them less than the non-authenticated ones. I, therefore, focus here on protecting the parts of the gateway, that are exposed to non-authenticated users.


DOS-Protection for the gateway, general concept and concerns

Citrix ADC, and thus of course the gateway, is inherently very well protected against DoS/DDoS attacks. In many cases, further protection is not necessary at all. I see a necessity at most where attacks are very likely, i.e. in the military field, with governments and with large international companies, especially if they deal with things that enrage certain population groups (I’m thinking of covid deniers, environmentalists, animal rights activists and similar groups). Unfortunately, it is not possible to use the AppQoE feature for this, so we have to work with the rate-limiting feature.

Rate limiting is a very simple function, but it is robust and easy to use. In principle, we would simply count all requests to certain URLs. If there are too many, we block them using responder policies.

Furthermore, we have to distinguish between DoS and DDoS, i.e. between Denial of Service and Distributed Denial of Service attacks.

DoS

In a simple DoS (Denial of Service) attack, a single user tries to send as many requests as possible to overload his victim. So to detect DoS attacks, a table containing only the IP addresses is sufficient. If an IP address sends suspiciously many requests, we block it.

To set a limit, you need to find out how many calls are generated during a normal successful login and how many are generated during a failed one. I would set the limit so that the limit is not reached, even after 10 failed logins. Caution: If your users come from NAT environments, then hundreds of users may share the same IP address. Take this into account in your dimensioning!

DDoS

DDoS (Distributed Denial of Service) attacks are similar to DoS attacks, but we are dealing with multiple simultaneous attackers. They are usually zombie PCs that send requests controlled from a central location. In contrast to DoS attacks, the aim here is to block requests to certain URLs from a certain IP address when they exceed a certain level. In order to detect DDoS attacks, we must therefore create a table that only contains the IP address and the URL.

The beauty of this is that it is very easy to define limits: A user may access all URLs only once if he enters invalid passwords 3 to 5 times, but not more often.

Concerns

Up to this point, everything has been simple. But we open the door to another DoS attack: The Citrix ADC / NetScaler of course needs RAM to create these tables. It is therefore possible to perform a DoS attack against the Citrix ADC / NetScaler by querying random URLs at high speed. In this way, an attacker could increase the tables to such an extent that the Citrix ADC / NetScaler runs out of memory and crashes. Of course, we have to prevent this, otherwise, we would make things worse instead of better. We will use a responder policy for this.

So the whole thing stands or falls on our ability to keep the tables small.

How to recognise users already logged on

Of course, we don’t want our DoS protection to affect legitimate, logged-in users at all. Therefore, the policies should only apply until the login procedure is completed. But how do we recognise this? It is relatively simple: As soon as the login is completed, the Citrix ADC / NetScaler sends two cookies: NSC_AAA and NSC_VPNERR.

NSC_AAA is the actual session cookie for logged-in users. This is the cookie that is actually the issue.
NSC_VPNERR contains a status. I wrote about this recently.

In principle, it is sufficient to check whether the user has an NSC_AAA cookie. HTTP.REQ.COOKIE.CONTAINS("NSC_AAA").NOT. We have to use this policy expression with all our responder policies. It would make sense to create a named expression for this use case. The command would be: add policy expression ex_authenticated HTTP.REQ.COOKIE.CONTAINS("NSC_AAA").NOT. (I’m not using this expression in my examples)

You may create expressions from GUI navigation to AppExpert → Expressions → Advanced Expressions.


Keeping tables small

There are several tricks to keep the tables small. The most important one is probably not to fill any data into the table that does not belong there. And you can do that by writing a responder policy that drops every request that goes nowhere.
Another possibility is to exclude query strings. I mean, to prevent /url?xxx from being treated differently from /url?yyy. This is done by using HTTP.REQ.URL.PATH as a criterion instead of HTTP.REQ.URL.

A responder policy that only allows requests to URLs that really exist and must be accessible to unauthenticated users

Unfortunately, I cannot give you a list of these URLs because it depends on the theme you are using and your customisations. But fortunately, the policy is easy to create.

In any case, we must first create the list. I use a pattern set (mine is called gw_urls). Next, we need to create a policy that drops all requests that target URLs that are not in the list, only for users who are not yet logged in.

Creating the pattern set

If you want to use the GUI: The pattern sets can be found in the AppExpert area. It is an advanced topic, so I only show the command-line version here.

add policy patset gw_urls
bind policy patset gw_urls "/"

(Initially, I just permit the root URL)
In GUI, navigate to AppExpert → Pattern Sets

I suggest, dropping potentially malicious requests. However, during the initial phase, my policy won’t drop, but just log, so I use NOOP instead of DROP. I make my policy logging of course.

add audit messageaction log_blocked ERROR "\"URL_blocked: \" + HTTP.REQ.URL"
add responder policy res_pol_GW_URLs "\n\nHTTP.REQ.COOKIE.CONTAINS(\"NSC_AAA\").NOT && HTTP.REQ.URL.PATH.EQUALS_ANY(\"gw_urls\").NOT" NOOP -logAction log_blocked

In Gui, navigate to System → Auditing → Message Actions for the message action and AppExpert →Responder → Policies for the responder policy.

As you can see, I create the audit policy first. It will log URLs that trigger the policy. I assume that you are familiar with audit policies and how to enable “User Configurable Logging”. Read here, if you are not. User configurable logging needs to be enabled, or it won’t work.
Next, I create the policy, for the time being with the action NOOP, which should later restrict access, and bind the audit policy to it. At the moment, it will just log URLs that are not in the list. Bind this responder policy to the gateway with a priority of 100.

initial Citrix ADC / NetScaler logging for DOS protection

As you can see, all URLs are logged. Now you only have to add the URLs to the list and the first problem is solved. You may make your work a bit easier by extracting a “ready to use the NetScaler command set” from the output by using this shell script: tail -F /var/log/ns.log |grep URL_blocked | tail -F /var/log/ns.log |grep URL_blocked | awk '{ printf "add policy patset gw_urls \""; print $18 }'

initial Citrix ADC / NetScaler logging for DOS protection

If we know all the URLs, we just have to bind them to the list like this:

bind policy patset gw_urls "/logon/LogonPoint/plugins/ns-gateway/nsg-epa.js"
bind policy patset gw_urls "/logon/LogonPoint/plugins/ns-gateway/nsg-setclient.js"
bind policy patset gw_urls "/logon/LogonPoint/plugins/ns-gateway/ns-nfactor.js"

Of course, you may also do this from GUI. Go to AppExpert → Pattern Sets, open the pattern set you previously created and add these URLs. Don’t forget about the / at the beginning of the URL!

I would strongly advise repeating the attempt several times so that all URLs are really on the list. Don’t forget to clear the browser cache in between. Repeat the attempt with the Workspace App (Citrix Receiver, PN Agent, …).


DoS-Protection

The rate-limiting feature requires a selector (what is counted?) and a limit identifier (how many hits can the entry get?). The result is then queried in a (responder) policy.

The Limit Selector

We can use the built-in selector Top_CLIENTS. The criterion is CLIENT.IP.SRC. Limit Selectors can be found and created at AppExpert → Rate Limiting → Selectors.

The Limit Identifier

The Limit Selector determines how many requests are allowed at which time. Limit Identifiers can be found and created at AppExpert → Rate Limiting → Limit Identifiers.
As I wrote above: Don’t forget that in NAT environments multiple users can use the same IP address. Many mobile networks NAT their users to the internet! It is better to set the limits too high than too low because the gateway can withstand a lot of requests.

add ns limitIdentifier sel_dos -threshold 50 -timeSlice 20000 -selectorName Top_CLIENTS
threshold: How many requests are allowed
timeSlice: In what time frame are the requests to be allowed?
selectorName: Which limit selector should be used

Because DOS attacks are only successful if they are very fast, it is sufficient if we set the limit rather short. My example allows 50 requests in 20 seconds (20,000 milliseconds). My example would allow two to three users from a NAT environment to log in within 20 seconds.

The responder policy

Like any policy that serves security, this policy should of course also log. So I create the log-action again first.

add audit messageaction log_DoS ERROR "\"DoS, IP blocked: \" + CLIENT.IP.SRC"
add responder policy res_pol_DoS "HTTP.REQ.COOKIE.CONTAINS(\"NSC_AAA\").NOT && SYS.CHECK_LIMIT(\"sel_DoS\")" DROP -logAction log_DoS

The responder policies in GUI can be found at AppExpert →Responder → Policies

This policy has to be bound to the gateway again, behind the policy I described above (otherwise the first policy has no effect anymore).


DDoS Protection

The DDoS policy is similar to the DOS policy, so I will only discuss the differences.

The Limit Selector

This time we have to create an identifier that contains both the IP address of the user and the URL.

add stream selector sel_DDoS HTTP.REQ.URL.PATH CLIENT.IP.SRC

As I mentioned before: I will just use the URL, not the query string. That’s why I use HTTP.REQ.URL.PATH instead of HTTP.REQ.URL. Again, I do this to keep the list short.

The Limit Identifier

add ns limitIdentifier sel_DDoS -threshold 15 -timeSlice 5000 -selectorName sel_DDoS

Again, don’t forget about NAT environment, a single IP does not mean a single user. Keep timeSlice short, don’t forget about false logons. My example would allow for about 3 users within 5 seconds.

The Policy

add audit messageaction log_DDoS ERROR "\"DDoS, URL blocked: \" + HTTP.REQ.URL"
add responder policy res_pol_DDoS "HTTP.REQ.COOKIE.CONTAINS(\"NSC_AAA\").NOT && SYS.CHECK_LIMIT(\"sel_DDoS\")" DROP -logAction log_DDoS

Bind this policy last.


I know, of course, that this is a very sensitive issue. I would be happy if you would discuss it with me! Any suggestion or criticism is welcome. Of course, I will mention you if you can contribute anything substantial.

About the author

Johannes Norz

Johannes Norz is a Citrix Certified Citrix Technology Advocate (CTA), Citrix Certified Instructor (CCI) and Citrix Certified Expert on Application Delivery and Security (CCE-AppDS).

He frequently works for Citrix international Consulting Services and several education centres all around the globe.

Johannes lives in Austria. He had been borne in Innsbruck, a small city (150.000 inhabitants) in the middle of the most beautiful Austrian mountains (https://www.youtube.com/watch?v=UvdF145Lf2I)

3 comments

    • Sure it is. Just add something like “CLIENT.IP.SRC.EQ(1.2.3.4).NOT” or “CLIENT.IP.SRC.IN_SUBNET(10.13.0.0/16).NOT” to your policy expression. That would exclude this IP or range of IPs.

      This expression is simpler than the rate-limiting one, so the would look like that:
      add responder policy res_pol_DoS “HTTP.REQ.COOKIE.CONTAINS(\”NSC_AAA\”).NOT && CLIENT.IP.SRC.IN_SUBNET(10.13.0.0/16).NOT” && SYS.CHECK_LIMIT(\”sel_DoS\”)” DROP -logAction log_DoS

Recent Posts

Recent Comments