Configuring Kerberos/AD Authentication For The SMB Server

By default the JFileServer SMB server uses NTLM authentication, for more secure and single signon authentication you can configure the SMB server to use Kerberos logons.

This document details how to configure the JFileServer, and JFileServer Enterprise, SMB server to use Kerberos authentication using a Windows Active Directory (AD) server setup.

Setting Up The File Server User Account

The JFileServer SMB server requires an AD user account to map a service principal name to that the client will request when accessing the SMB server.

Create a user account on the AD server, fill in the user logon name eg. jfileserver, disable the 'User must change password at next logon' option, and enable the 'Password never expires' option. Set a password for the new account.

In the user account properties enable the 'Do not require Kerberos pre-authentication' option. If there are options for AES 128 bit and AES 256 bit encryption then enable both options.

The file server host also requires a DNS entry with the AD server.

Note: If you are using an older Java VM that does not have the high strength encryption policy enabled then do not enable AES 256 bit encryption for the account. This may be earlier versions of Java 8 or pre Java 8 versions. If in doubt then do not enable AES 256 bit encryption. Only Windows 11 and the latest macOS clients support AES 256 encryption, and JFileServer does not currently support or negotiate AES 256, it will negotiate to use AES 128 only.

Generate The Keytab and Service Principal Name

Generate a key table for the file server using the ktpass command.

ktpass -princ cifs/<file-server-host-name>.<domain>@<REALM>
 -pass <file-server-account-password>
 -mapuser <domain>\<file-server-account-name>
 -crypto ALL
 -ptype KRB5_NT_PRINCIPAL
 -out <output-file-name>
 -kvno 0

If you only want to use a particular encryption type then use -crypto AES128-SHA1 or -crypto AES256-SHA1. The older encryption setting of -crypto RC4-HMAC-NT can be used but may not allow some newer clients to access the file server.

The generated key table will be used by the SMB server to authenticate the server so it will then be able to authenticate the client logons.

Copy the key table to the file server host system.

Create the service principal name mapping, that clients will use to connect to the SMB file server service, using the setspn utility. Newer versions of the ktpass command will set the SPN when generating the key table.

To check if the SPN is set use

setspn -Q cifs/<file-server-host-name>.<domain>

If the SPN has not been set then use the following commands to set the SPNs

setspn cifs/<file-server-host-name> <file-server-account-name>
setspn cifs/<file-server-host-name>.<domain> <file-server-account-name>

Configure Kerberos On The File Server

The Java authentication APIs require a Kerberos configuration file, this can either be in the default location such as /etc/krb5.conf on linux and macOS, C:\winnt\krb5.ini on Windows, the location can be specified on the Java command line using the java.security.krb5.conf property, or using the JFileServer configuration value <KerberosConfig> to specify the configuration file path and name.

A sample Kerberos configuration file is shown below, the Kerberos realm is always specified in uppercase.

[libdefaults]
default realm = FILESYS.ORG

[realms]
FILESYS.ORG {
  kdc = adsrv.filesys.org
  admin_server = adsrv.filesys.org
}

[domain-realm]
adsrv.filesys.org = FILESYS.ORG
.adsrv.filesys.org = FILESYS.ORG

Setup The Java Login Configuration

The Java login configuration is used by the file server application to logon the server application using Kerberos by using the details in the key table that was generated on the Windows AD server.

The Java login configuration file can be stored in your user home folder or a system folder such as /etc. Create a new file called jfileserver.login.conf. The default Java login configuration section name is FileServerSMB. A sample Java login configuration file is shown below :-

FileServerSMB {
 com.sun.security.auth.module.Krb5LoginModule required
 debug=false
 storeKey=true
 useKeyTab=true
 keyTab="<path-to-the-key-table-file>"
 principal="cifs/<file-server-host-name>.<domain>";
};

Note: Use spaces in the Java login configuration file, tab characters cause problems.

To enable the Java login configuration use the JFileServer configuration value <LoginConfig> to specify the path to the login configuration file.

You can also set the login configuration file by updating the main Java security configuration file. This is usually located in the <JRE-folder>/lib/security/java.security file.

Look for the example configuration line that starts with #java.login.config.url.1=.... Uncomment the line, or add a new line, similar to the value shown:-

java.login.config.url.1=<path-to-the-java-login-configuration-file>

Configure The JFileServer SMB Authentication

In the JFileServer XML configuration file enable the SMB server Kerberos authentication via the <authenticator> configuration section within the <fileserver> <SMB> section of the configuration. A minimal sample configuration section is shown below:-

<fileserver>
...
  <SMB>
  ...

    <authenticator type="enterprise">
      <Realm>FILESYS.ORG</Realm>
    </authenticator>
  </SMB>
  ...
</fileserver>

The following configuration tags are available for the SMB authenticator:-

Configuration Value	Description
<useSPNEGO/>	Force Simple Protected Negotiation to be used for the authentication, this will be selected automatically if Kerberos authentication is enabled. If only NTLM authentication is enabled the default is to use NTLMSSP in the authentication exchanges.
<mode>...</mode>	Server mode, should always be 'USER'
<Realm>...</Realm>	Kerberos realm, always uppercase
<LoginEntry>...</LoginEntry>	Specifies the Java login configuration entry name if the default name of 'FileServerSMB' is not used
<Password>...</Password>	Kerberos file server account password. Not required
<disallowNTLMv1/>	Do not allow the weaker NTLM v1 logons, only NTLM v2 logons will be accepted
<disableNTLM/>	Only allow Kerberos authentication. NTLM authentication will not be advertised by the server or accepted
<Debug/>	Enable authentication debug output
<kerberosDebug/>	Enables the Java authentication API debug output. Equivalent of setting the Java system properties 'sun.security.jgss.debug' and 'sun.security.krb5.debug'
<KerberosConfig>...</KerberosConfig>	Specifies the location and name of the Kerberos configuration file
<LoginConfig>...</LoginConfig>	Specifies the location and name of the Java login configuration file