Trouble shooting Citrix NetScaler Gateway connection issues

One of the most annoying issues in Citrix NetScaler are ICA / HDX connection issues. The reason for this is the way connection issues are reported.

There are two potential sources of trouble: Citrix StoreFront and Citrix NetScaer Gateway. So I will divide my blog in three sections: How to find the source of trouble, Trouble shooting Citrix StoreFront and Trouble shooting Citrix NetScaler Gateway.

How to find the source of trouble

It seems to be annoying and hardly possible. I am one of the moderators of a Facebook group about Citrix. Questions about connection issues come up quite often. Most of the answers don’t focus to the right source. They hardly ever ask: Which component is guilty. Instead people give misleading tips. I want to keep away from misleading tips, instead guide you through a well structured trouble shooting guide.

Let’s try to understand what’s going on:

The stages of a Citrix NetScaler Gateway connection

I talk about using Citrix StoreFront website, there is not so much difference to a receiver for web site. If you still use Citrix WebInterface: not much difference there, but my screen shots won’t be of any help.

  1. a user connects to the NetScaler Gateway website and is prompted with a logon page
  2. the user enters his credentials. This credentials are checked against logon providers like LDAP and RADIUS based sources (Active Directory, RSA, Safe Word, SMS Token and many more).
  3. The user will see applications only after logging on successfully. So logon is done and without any issue as soon as we see applications!
    We now know: NetScaler Gateway was able to authenticate the user, it also connected to Citrix StoreFront (or Web Interface) successfully and StoreFront was successful connecting to XML broker service.
    So no need to check here, it’s already checked: Logon works perfectly fine, the connection to StoreFront / Web Interface worked fine, and it’s connection to XML broker service is tested (we would not see any application if any of them fails)
  4. The user clicks an application. This click is proxied via NetScaler Gateway and StoreFront (WI) to XML broker service. XML broker service selects a resource, a desktop or an application, connects to this resource’s IP vis HTTP(s) (XenDesktop) or IMA (XenApp up to version 6.5), and stores this user’s credentials inside this machine. The machine returns a so called NFuse ticket (NFuse is the old name of Citrix Web Interface). The IP address together with this NFuse ticket is returned to StoreFront (Web Interface).
  5. Getting an STA ticket: This is a first source of problem I want to go into: We have to store the target’s IP address inside our secure environment. The store we use is called STA, and it’s usually one of the XenApp servers or XenDesktop DDCs (desktop delivery controler). The STA returns a so called STA ticket.
  6. We now create an ICA file. The ICA file will contain the name of the NetScaler Gateway (FQDN), the NFuse ticket and the STA ticket (don’t mix these up!) together with some information about screen resolution, clip board mapping and so on. I attached a sample ICA file:
    [ApplicationServers]
    Notepad=
    ...
    [Notepad]
    Address=;40;STA324731891;832A84599E0B7449B8578DCB8DBA95  this is STA ID and STA ticket
    AutologonAllowed=ON
    BrowserProtocol=HTTPonTCP
    CGPSecurityTicket=On
    ClearPassword=E16458A6937769  This is the 1st half of the NFuse ticket
    ...
    Domain=\C48CC641E8301B33  This is the 2nd half of the NFuse ticket
    ...
    InitialProgram=#Notepad
    ...
    Launcher=WI
    ...
    LogonTicket=E16458A6937769C48CC641E8301B33  this is the NFuse ticket
    LogonTicketType=CTXS1
    ...
    SSLProxyHost=gateway.norz.at:443  The FQDN of the NetScaler Gateway used by Receiver
    ...
    TransportDriver=TCP/IP
  7. This ICA file is returned to the client via NetScaler Gateway. We don’t need to consider this connection to be guilty for our problems as it already tested: it worked fine before!
  8. The browser forwards this ICA-File to the Citrix receiver. (Begin of second part!) Citrix receiver will read the ICA file and
  9. connect to the NetScaler Gateway. We can see this as we will see a progress bar.
  10. The receiver will send the STA ticket to the NetScaler Gateway. NetScaler Gateway will connect to the STA and try to resolve this ticket.
  11. As soon an NetScaler Gateway was able to resolve the ticket, NetScaler Gateway will try to connect to the target device (XenApp server, VDI devices)
  12. the application / desktop launches.

It’s essential to understand the connection process you want to trouble shoot!

So, where does it break into parts?

I have already mentioned: as soon as the ICA file is created and returned to the client the second part starts. How can we find out? Easy like that: The Citrix receiver (former names: ICA- client, ICA plugin, Citrix client, and approx 1.742.946 names more) is started, we successfully passed the first stage. So this is my first question: did it download the ICA file?

No,
I did not download an ICA file
so let’s continue trouble shooting Storefront!
Yes
I downloaded an ICA file
so let’s continue trouble shooting NetScaler gateway!

You are not sure if you received an ICA file or not?

  • Firefox: The ICA file goes into your download area, typically %username%\AppData\Local\Temp (or %tmp%). However it usually gets deleted immediately.
  • Internetexplorer: There is a file created in %tmp%, but it is not accessible, it’s extension is not .ICA. However it usually gets deleted immediately.
  • Chrome: It’s the same: the file goes to %TEMP%. Thanks, Hendrik Klinge for this information! It is unchanged, so more or less the same as Firefox.

As the ICA file usually gets deleted immediately you may use Microsoft’s Process Monitor to be 100% sure! You could also edit the ICA file in StoreFront (C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica). It is a windows INI file. You may change RemoveICAFile=yes to no in [WFClient] section, so it will stay for ever (and spam the %tmp% directory).

More methods to find the stage of the connection process

Usually you will see an error message. It’s stage 1 (StoreFront alone to blame for your issue) if this error message is displayed inside your browser, it’s stage 2 if it’s a windows (Mac, Linux, …) message box.


Trouble shooting Citrix StoreFront

If you got stuck within the first portion of the connection process, your issue is not directly related to NetScaler, you don’t even need to log on to NetScaler!

  1. Log on to your StoreFront server and check NetScaler Gateway settings:
    • Your authentication methods have to contain Pass-Through from NetScaler Gateway (right hand side, lower section, Manage Authentication Methods)

      Pass-Through from NetScaler Gateway

      StoreFront: Pass-Through from NetScaler Gateway

    • You need to define a NetScaler Gateway (right hand side, upper section, Manage NetScaler Gateways)
      Storefront: set a NetScaler Gateway

      Storefront: set a NetScaler Gateway

      Storefront: set a NetScaler Gateway

      Storefront: set a NetScaler Gateway, Detail

      Don’t check authentication settings: Authentication worked fine, so there is nothing to do in here!

    • Also check the STAs. The STAs have to be resolvable! (same dialogue as above)
      STA settings in Citrix StoreFront

      STA settings in Citrix StoreFront

      Use telnet (or putty) to connect to the desired port. So in my example I would do a telnet XD7-DC.norz.local 80. The screen will turn black if it is able to connect. If I enter “something” I will see some HTML output. I won’t see anything if I connect to an https based server: telnet XD7-DC.norz.local 443 as I won’t be able to do a SSL hand shake. If you mistyped the name of the STA, or the STA is not reachable you will see:
      telnet XD7-DC.norz.lokal 443
      connecting to XD7-DC.norz.lokal…
      .
      The connection attempt will time out. Always do this tests from your StoreFront servers!
      Reasons for a STA not being reachable may be a miss-typed STA name or the (application) firewall blocking connections.

    • Enable remote access! (right hand side, lower section, Configure Remote Access Settings).

      Enable remote access in StoreFront

      Enable remote access in StoreFront

  2. There should not be the need to mention, as this is very basic windows administration strategy, however I see tons of people not being aware of it: Check the event log of your StoreFront servers!

Events and their meanings

If something goes wrong in StoreFront you usually see this message:

StoreFront problem

There is something wrong in StoreFront

you will know: We never downloaded an ICA file. We are in trouble with StoreFront. Never check Citrix NetScaler Gateway, it was not involved, check events in StoreFront server. It may be hard to locate an event if you load balance your StoreFront servers, so I tend to disable all services but one.

Events pointing to STA problems:

The events can be found, both in administrative events, or in “Application and Service Logs” -> “Citrix Delivery Services”

There will be a set of events: Citrix Store Service, Error 0, Citrix Store Service, Error 1003, Citrix Store Service, Warning 28.

Citrix Store Service, Error 0, 1003, Warning 28

Events related to STA problems

Citrix Store Service Error 0, Citrix StoreFront: wrong STA name or STA not reachable

Citrix Store Service Error 0

Store Service Error 0: The server name <your server’s name> cannot be resolved. The specified Secure Ticket Authority could not be contacted and has been temporarily removed from the list of active services.

I think, the meaning of this event is more than clear: Citrix StoreFront could not connect to at least one of the STA servers you specified. There might be a chance to connect if there is more than a single STA server. Anyway: You should fix this problem!

Citrix Store Service, Error 1003: No STA server available!

Citrix Store Service, Error 1003 event

Citrix Store Service, Error 1003. All the configured Secure Ticket Authorities failed to respond to this XML transaction: http://<yor server mane>/scripts/ctxsta.dll.

This event will always follow one or more Citrix Store Service, Error 0 events. This is a serious event, it means: It’s absolutely impossible to launch an application or desktop: There is no STA server available. Citrix Store Service, Error 1003 has to be fixed, it’s the reason for your connection problems! No way: You have to fix this problem!

Citrix Store Service, Warning 28

Citrix Store Service, Warning 28: Failed to launch the resource ‘Local.<your application/desktop name>’, unable to obtain a ticket from the configured Secure Ticket Authorities.

This is the final result. We could not launch the application. It’s just a summary, fix Citrix Store Service Error 0 above and you’ll get rid of the 1003 and this one at the same time!


Trouble shooting Citrix NetScaler Gateway

Our problem is related to NetScaler Gateway if we successfully mastered part 1. So let’s trouble shoot problems here.

Citrix Receiver was unable to connect via NetScaler Gateway

Unable to launch an application. Contact your helpdesk with following information: Cannot connect to the Citrix XenApp server. …

Before we see an error like this we will see the progress bar indicating: Citrix Receiver received a STA file. This progress bar is of some interest! Unfortunately this message may disappear way too fast, so you will probably just see the message above.

Citrix receiver launching an application

Citrix receiver launching an application

That’s an absolutely thrilling information for all of you! Click on “more information” and you’ll see where we got stuck!

Citrix receiver connecting to NetScaler Gateway

Citrix receiver connecting to NetScaler Gateway

So this picture shows the receiver establishing a connection to Citrix NetScaler Gateway. To be 100% clear: we still are not connected! We are just establishing a connection to NetScaler Gateway, so a TCP Sync packet is sent, but the TCP/IP connection is either still not established, or the SSL connection is not established yet!

Reasons for connections failing during this stage:

There may be several reasons for connections failing during this stage:

  1. the name of the gateway can’t get resolved. Check the name in StoreFront.
    StoreFront: set a NetScaler Gateway

    Storefront: set a NetScaler Gateway

    Storefront: set a NetScaler Gateway, Detail

  2. The Citrix NetScaler Gateway server certificate is not trusted, or the certificate chain is broken. So as the first step: download NetScaler Gateway’s certificate and open it at your workstation (not in a browser, just from OS). Resolve all problems with this certificate. Don’t even think of continuing without solving this problems, it doesn’t make any sense at all.
  3. If you miss the intermediate CA certificate you have to download it and import it into NetScaler and link it.
    NetScaler 11.1: Go to Traffic Management → SSL → CA Certificates. Import the certificate. Next go to Traffic Management → SSL → Server Certificates. Click the NetScaler Gateway server certificate. Than Action and Link. It should display the certificate of the intermediate CA. Click OK.

STA Tickets

So we successfully connected to Citrix NetScaler Gateway. Connection in progress disappeared. The current state is connected: There is a SSL connection from Client to NetScaler Gateway.

During the next stage the Citrix receiver will send the STA ticket to NetScaler Gateway, and it will try to resolve the STA ticket. To do so it has to connect the configured STA.

STAs don’t replicate (actually they don’t even know about each other), so we need to specify exactly the same STA to NetScaler Gateway as we did in StoreFront. We will have to check StoreFront for STAs (see here). We then will check Citrix NetScaler Gateway for STA settings.

We navigate to NetScaler Gateway → Global Settings:

Seting STA in NetScaler Gateway

Seting STAs in Citrix NetScaler Gateway

NetScaler Gateway: STA down, no connection possible

NetScaler Gateway: all STAs are down, so no connection possible

As you see: the bound STA appears to be down. There are 3 reasons for this:

  1. the name is wrong, or can’t get resolved. I would put the name into clipboard and then navigate to System → Diagnostics and start the ping utility. Paste the host name into clipboard and see if it is ping-able. You will see, at least, if the host name is resolvable
  2. the host name is not resolvable. So the DNS server you configured for your NetScaler gateway is unable to resolve the host name. In both cases the result of this ping will look like that:

    NetScaler Gateway: STA hostname not resolvable

    NetScaler Gateway: STA host name not resolvable

  3. a firewall is blocking the STA communication.

After resolving all of these issues the STA settings in NetScaler Gateway should look like this:

NetScaler Gateway: All STA servers are up now

All STA servers are up now

You will notice the STA IDs, indicating NetScaler Gateway could connect to this STA at least once, and the green light (it may be missing with some elder versions of NetScaler) indicates actual connections.

No more problems about NetScaler Gateway and StoreFront as soon as you are fine until here!

It takes too much time to establish connections from outside, compared to inside?

Don’t blame NetScaler for this:

So NetScaler knows where to connect. NetScaler will use TCP/2598 for this connection: CGP (Citrix Gateway Protocol, former name: Common Gateway Protocol). At least as long as you did not turn off session reliability. I bet my life, you did not. NetScaler Gateway will try to connect via CGP for 30 seconds, than give up and try plain HDX (formerly known as ICA) on TCP/1494. So open up TCP/2598 on your firewall, it will safe you 30 valuable seconds!

Your connections still fail?

Let’s keep thinking: we successfully connected to NetScaler Gateway. We successfully resolved the ticket, so NetScaler Gateway now connects to the target device: a Citrix XenApp server or a Citrix XenDesktop VDI device.

So there are 2 reasons for this issue:

  1. a firewall blocks the connection
  2. NetScaler Gateway does not know a route to this IP

Just resolve these issues by opening up the firewall ports, or add the route to the desired network.


I hope this helped! Feel free to ask if you see additional problems not covered in here, I’ll answer your question and add the solution here.

Unfortunately I was unable to capture screen shots from Citrix Receiver connection stages due to my (relatively) fast environment. I’d be glad to get your screen shots 😉

8 thoughts on “Trouble shooting Citrix NetScaler Gateway connection issues

  1. Hendrik Klinge

    >Chrome: I don’t know where chrome puts these files too! (tell me if you know)
    I use up-to-date-Chrome on up-to-date Win10. And I connect to StoreFront. My ICA files go into %TEMP%. I manually delete them from time to time. Sample file name : “ica5048260177296.ica”

    Reply
    1. Johannes Norz Post author

      Thanks, Hendrik, so they all do more or less the same. Yes, some times they don’t get deleted nowadays. I guess it’s a bug in Citrix Receiver. The template ICA file is in C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica. It is a windows .ini file and contains (per default) a RemoveICAFile=yes in [WFClient] section, so it should definitely get deleted :-).

      Reply
  2. Jo Palmer

    I get through all the stages until I click on an application and then I get a message that tells me it took too long to connect? and asks me to check Proxy settings and firewall settings.
    The connection worked perfectly a few days prior, so why have things changed?
    And what can I do about it. I”m an Australian working in London.

    Reply
    1. Johannes Norz Post author

      looks like this client can’t connect to your NG. There are several reasons. Did you update your certificate to SHA2? if so you will have to upgrade your receiver.

      Reply
  3. Andrew Bradley

    Hi

    We are having an issue similar to this in our environment, except it is when pulling an active internal session to external. It seems to work intermittently. Applications can be successfully launched by clicking the icon but this opens a seperate connection to another server.

    Any ideas?

    Reply
    1. Johannes Norz Post author

      This may happen if you did not add a call back URL in StoreFront. Check if you added a callback URL. An other potential reason would be XML Service Trust

      Reply
  4. Andrew Bradley

    Thanks for the quick response! Call back url is present and has been working. We have had this issue for a long time. Citrix recommended we enable ica proxy migration on gateway vip but this has made no difference. I have tested in both basic and smart access mode. It”s strange that it works randomly, i can see reciever start to load the active apps but it disconnects quickly or hangs when negotiating.

    I am not sure what the xml service trust is. Where can i find this setting? I appreciate the help 🙂

    Reply
    1. Johannes Norz Post author

      Hm, Andrew, seems to be not that easy 🙂

      I don”t think it”s a matter of trust, (https://docs.citrix.com/en-us/xenapp-and-xendesktop/xenapp-6-5/xenapp65-w2k8-wrapper/xenapp65-admin-wrapper/ps-maintain-wrapper-v3-tsk/ps-sf-citrix-xml-service-port-set-v2-tsk.html) any more.

      The session is detected (and therefore disconnected from current Client, if it was active before). Next it should get reconnected, it gets reconnected, but it fails during the “negotiating catabilities” state? Is this true?

      To be honest, I have no idea. I would do a Network trace, just to see what”s going on. But I”m pretty sure I would not find out.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *