Importing an existing NetScaler configuration from MPX/SDX into a VPX

I

It’s one of the things I do most often: Import a customer’s NetScaler installation into a VPX, so I can have a look at it. In fact, I do most of my audits that way.

What is required

The best basis would be a Citrix NetScaler backup. It contains everything we need, except for certificates, together with some other data in addition, like logs. The ns.conf file, alone, would be insufficient in many cases!

  • the ns.conf file from /flash/nsconfig
  • all certificates from /flash/nsconfig/ssl  (my customers, however, don’t like to give certificates to me, see here)
  • everything from /var/download (it contains WAF and BOT signatures, imported text files for error messages and many more)
  • everything from /var/netscaler/logon (it contains UI customization for AAA and gateway, logon field definitions and many more)
  • the location database from /var/netscaler/locdb (only if location based oplicies are in use)
  • gslb definitions from /var/netscaler/gslb and /var/netscaler/gslb_sync (only if GSLB is in use)
  • everything from /var/netscaler/gui (GUI customizations for VPN, endpoint analysis scans, …)

In addition, I would usually need to know the nsroot password, but some of my customers don’t want to handle it over to me. So I have to hack their nsroot account (and other accounts of interest)

Creating the VPX

Import a proper VPX image to the hypervisor of your Choice. I mainly use KVM, but Citrix Hypervisor, VMWare ESX or workstation, or even Microsoft’s HyperV would do. The VPX should have the same amount of network cards, as the original Citrix NetScaler. The amount of RAM should be 4 GB at least. Install this VPX and do basic configuration (IP, gateway). Connect there and backup the ns.conf file.

Changes to the ns.conf file

There are some changes you have to do to the ns.conf file.

Network cards:

The name and type of network cards will be different between a VPX and an MPX.

This is a typical network card configuration from an MPX

set interface 100/1 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/1
set interface 100/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/1
set interface 100/3 -lacpMode ACTIVE -lacpKey 2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/2
set interface 100/4 -lacpMode ACTIVE -lacpKey 2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/2
set interface LO/1 -haMonitor OFF -haHeartbeat OFF -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype Loopback -ifnum LO/1

This is a part of the configuration of a customer’s NetScaler MPX. They have 4 network cards, all four are parts of link aggregation. The name of the interfaces is 100/1 (the 1st 100 GBt interface) to 100/4 (the 4th one). The first part if the name is the speed (in this case, 100 GBt) the second one is the number of the interface. The type of interfaces (intftype) may vary, depending on NetScaler model. In this case, 100/1 and 100/2 are in the link aggregate LA/1 (ifnum), 100/3 and 100/4 are in a link aggregate LA/2 (supporting LACP) as well.

This is a typical network card configuration from a VPX

set interface 1/1 -state DISABLED -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Xen Virtual” -ifnum 1/1
set interface 1/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Xen Virtual” -ifnum 1/2

All interfaces of a VPX are 1 GBt only. That does not necessarily mean, they are limited to 1 GBt, speed depends on the hypervisor. Type of interface always is “Xen Virtual”

In this case, the two interfaces are sufficient to migrate, as the original MPX just has two logical interfaces.

Adaptions to the ns.conf

Link aggregation is not supported by the hypervisor, so we need to remove the definition of link aggregation.

0
# grep LA/ ns.conf.ns4
1
set interface 100/1 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 –intftype “Mellanox 100G” -ifnum LA/1
2
set interface 100/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 –intftype “Mellanox 100G” -ifnum LA/1
3
set interface 100/3 -lacpMode ACTIVE -lacpKey 2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/2
4
set interface 100/4 -lacpMode ACTIVE -lacpKey 2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Mellanox 100G” -ifnum LA/2
5
add channel LA/1 -lamac b2:22:40:b6:fd:a2 -ifAlias VLAN920 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -devno 709
6
set channel LA/1 -lamac b2:22:40:b6:fd:a2 -ifAlias VLAN920 -throughput 0 -lrMinThroughput 0 -bandwidthHigh 0 -bandwidthNormal 0
7
set channel LA/2 -lamac 86:aa:99:1b:0a:e0 -ifAlias VLAN918 -throughput 0 -lrMinThroughput 0 -bandwidthHigh 0 -bandwidthNormal 0
8
bind channel LA/1 1/1 -devno 709
9
bind channel LA/1 1/2 -devno 709
10
bind vlan 4000 -ifnum LA/2 -tagged

I replaced 100/1 with 1/1, “Mellanox 100G” with “Xen Virtual” and the ifnum with 1/1, as the configuration has to run on a VPX in line 1.
I deleted line 2.
I replaced 100/3 with 1/2, “Mellanox 100G” with “Xen Virtual” and and the ifnum with 1/2 removed references to LACP in line 3.
I deleted lines 4 to 9.
I replaced LA/2 with 1/2 in line 10, as they deal with link aggregation, which won’t work in a VPX.
That’s this part of ns.conf in the final ns.conf:

set interface 1/1 -state DISABLED -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Xen Virtual” -ifnum 1/1
set interface 1/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype “Xen Virtual” -ifnum 1/2
bind vlan 4000 -ifnum 1/2 -tagged

So I won’t expect any troubles with a badly migrated network configuration.

Users

This is a typical user configuration:

add system user myuser 5deb3d06742ebf84ad1b…1745ce5be4a337 -encrypted

there is a name (myuser) a password (I shortened it down) and the parameter -encrypted, meaning, this password string password is not plain text. The password, however, could also be in plain text, if you delete -encrypted. And that’s what I usually do:

add system user myuser Password1

My new password will be Password1. And it will get encrypted after the next “save configuration”.

Faking Certificates

Well, it’s sad but true. Customers usually don’t trust me that far. And that’s right, there is absolutely no need to handle certificates together with the private keys to anybody outside the NetScaler team. So I usually don’t get these certificates. That’s not a big problem. I could skip certificates, this won’t break the deployment but causes vServers to be down, or I can simply fake their certificates. To be able to do so, I have to find the names of the certificate- and key files.

Finding certificate names

I execute the following program: grep "add ssl certKey" ns.conf. The output will look like that:

add ssl certKey wonderkitchen.network -cert wonderkitchen.network.cer -key wonderkitchen.network.key 2ce1ce11e….744c9c50fa4d -encrypted -encryptmethod ENCMTHD_3 -kek -suffix 2021_03_11_18_55
add ssl certKey wildcard.norz.at -cert wildcard.norz.at-23.crt -key wildcard.norz.at.key c0036394…28f54d3f590ad -encrypted -encryptmethod ENCMTHD_3 -kek -suffix 2021_03_11_18_55
add ssl certKey digicert_intermediate -cert intermediate.cer

You can see, there are three certificates and an intermediate CA certificate. I’m not interested in this intermediate certificate, to be honest. I have to create 3 private keys, called wonderkitchen.network.key, wildcard.norz.at.key.

If I fake certificates, I can also remove lines like this one

link ssl certKey wonderkitchen.network digicert_intermediate

(it links the CA-Certs to certificates)

Creating Keys, CSR and certificates

Faking certificates should be done on the newly imported VPX. If you do it there, certificates will already be in place, right when you boot the box after importing the ns.conf.

Keys get created on a NetScaler easily. This creates a wonderkitchen.network.key keyfile:

create ssl rsakey wonderkitchen.network.key 2048 -password 12345

 

Next step: creating a CSR. The name of the CSR does not matter, I follow my naming convention and name it similar to the certificate file:

create ssl certReq wonderkitchen.network.csr -keyFile wonderkitchen.network.key -countryName at -stateName Tirol -organizationName norz.at -subjectAltName colors.workspacelab.com

 

Creating the certificate. This example creates a certificate file called wonderkitchen.network.cer.

create ssl cert wonderkitchen.network.cer wonderkitchen.network.csr SRVR_CERT -days 1825 -CACert ns-root.cert -CAKey ns-root.key -CASerial ns-root.srl

 

IP addresses

Sure, IP addresses have to get adapted in most cases. If I do an audit, I don’t change IP addresses, instead, I create a new SNIP in the local network. That’s how IP addresses look like in a NetScaler definition:

set ns config -IPAddress 192.168.229.10 -netmask 255.255.255.0
add ns ip 192.168.0.15 255.255.255.0 -vServer DISABLED -telnet DISABLED -ftp DISABLED -gui DISABLED -snmp DISABLED -mgmtAccess ENABLED -networkRoute ENABLED
add ns ip 192.168.200.15 255.255.255.0 -vServer DISABLED -mgmtAccess ENABLED

If you change IP addresses, make sure that all occurrences of those IP addresses are changed as well. I would suggest doing a search and replace so there are no orphaned remnants of the original IP.

If I want to create a SNIP address in my local network, I would add the following line:

add ns ip 10.0.0.10 255.255.255.0 -vServer DISABLED -telnet DISABLED -ftp DISABLED -snmp DISABLED -mgmtAccess ENABLED

I would use this IP to connect to this newly imported VPX I would not be able to connect to the NSIP. I try to avoid changing routes.

IPs may also occur with vLan bindings, NAT, virtual servers, ACLs and some more.

Let’s give it a go!

We should be done. I’d strongly recommend doing a snapshot now. Upload this updated ns.conf file to the VPX. Delete all currently existing /var/log/ns.log files. Turn off the VPX (don’t shut it down to avoid logs that may origin from the shutdown) and boot it.

Connect via ssh (putty), as soon as the VPX is up and running. Download /var/log/ns.log. Look for error messages! It will rise error messages like

Apr 20 14:47:52 <local0.debug> vpx [1235]: nslocal_fileRead(): Data secured for:training.lab
Apr 20 14:47:52 <local0.debug> vpx [1235]: nslocal_fileRead(): Data secured for:training.lab
Apr 20 14:47:53 <local0.err> vpx nsconfigd: _dispatch(): Object not readable [Please make sure it exists].
Apr 20 14:47:53 <local0.info> vpx nsconfigd: _dispatch(): Failing command:add responder action res_act_error respondwithhtmlpage errormessage -responseStatusCode 200″
Apr 20 14:47:53 <local0.err> vpx nsconfigd: _dispatch(): Action does not exist
Apr 20 14:47:53 <local0.info> vpx nsconfigd: _dispatch(): Failing command: “add responder policy res_pol_error “HTTP.REQ.URL.EQ(\”/test\”)” res_act_error

What’s wrong here? In the third line, you see an object is missing. In the next line, we see that creating a responder action fails as well (because the missing object is the HTML page errormessage). So the action does not exist and creating the policy will fail as well.

 

 

About the author

Johannes Norz

Johannes Norz is a Citrix Certified Citrix Technology Advocate (CTA), Citrix Certified Instructor (CCI) and Citrix Certified Expert on Application Delivery and Security (CCE-AppDS).

He frequently works for Citrix international Consulting Services and several education centres all around the globe.

Johannes lives in Austria. He had been borne in Innsbruck, a small city (150.000 inhabitants) in the middle of the most beautiful Austrian mountains (https://www.youtube.com/watch?v=UvdF145Lf2I)

2 comments

  • Hi,
    a question. if i delete the set interface commands and dont replace them.
    Do the vpx takes the first configuration set interface settings?
    Best Regards

Recent Posts

Recent Comments