This blog is an addendum to my earlier post on enabling SSO for Splunkweb administration console. That post deals with only one splunkweb instance. What if customer wanted to enable SSO for farm of search heads running behind a load balancer. This deployment is popular with the new search head clustering feature introduced in Splunk 6.2 version.
In order to enable the splunk web SSO for the search head clustering deployment, you need to have access to a load balancer (hardware or software with the ability to route the requests based on Cookie or IP persistence ) that plays the vital role to achieve the SSO seamlessly among the backend search heads. This procedure has been tested with both hardware and software load balancers.
- Big IP with IP persistence – Hardware Load Balancer
- Apache httpd reverse proxy with sticky cookies. – Software Load Balancer
The configuration is simple
- Enable the SSO using the usual procedure as described here
- Create the LDAP authorization strategy in all the participating search heads in the search head clustering deployment. (You can simply copy the authentication.conf to all the search heads, make sure the bind user password is in plain text before copying, once you restart the splunkd these plain text passwords will be obfuscated )
- Set the load balancer configurations appropriately, especially the stickiness for the requests. If the requests ends up in the wrong server then you would see unexpected results.
Setting up Big IP LB for SHC
It is relatively easy to setup the sticky load balancing for the HTTP traffic. You can find documents related to this here
How to setup apache reverse proxy for SHC load balancing
In this section I am going to describe the procedure about the configuration of apache reverse proxy for load balancing along with cookie based persistence.
- Apache httpd server
Server version: Apache/2.4.10 (Unix)Server built: Jan 1 2015 13:25:30Server’s Module Magic Number: 20120211:36Server loaded: APR 1.4.6, APR-UTIL 1.4.1Compiled using: APR 1.4.6, APR-UTIL 1.4.1
- Required apache modules
- An LDAP server for authentication
What needs to be configured at Search Heads?
In the search heads that participate in the search head clustering, please enable the usual SSO configuration properties.
[settings] root_endpoint=/shc trustedIP = 127.0.0.1,10.140.104.34 remoteUser = X_Remote_User SSOMode = permissive embed_uri = http://proxy.sv.splunk.com:8080/gpass allowSsoWithoutChangingServerConf = 1 remoteUserMatchExact = 1
The above configuration has to be included in all the participating search heads in the search head clustering.
There are couple of new properties added in the Splunk enterprise 6.2
- remoteUserMatchExact – When matching the remoteUser header, consider dashes and underscores distinct (so “Remote-User” and “Remote_User” will be considered different
- allowSsoWithoutChangingServerConf – Usually when configuring SSO, a trustedIP needs to be set both here in web.conf and also in server.conf. If this is set to “1” then we will enable web-based SSO without a trustedIP in server.conf
- authentication.conf – Create the LDAP authorization strategy in all the participating search heads in the search head clustering deployment. (You can simply copy the authentication.conf to all the search heads, make sure the bind user password is in plain text before copying, once you restart the splunkd these plain text passwords will be obfuscated ). The LDAP server that is used in this configuration file must be the same as in the httpd.conf used for reverse proxy authentication.
Restart the whole cluster by issuing splunk rolling-restart shcluster-members
That is it, you are ready to setup the load balancing using the apache http server.
Configuring the Apache Web server as Load balancer
Once you configure the apache http server with required modules, you can go ahead configure the web server to act like a load balancer. Here are the relevant portion that needs to be configured for apache http server.
Add the following portion of configuration code that goes in the httpd.conf by-passes the authentication so the embedded reports can be accessed anonymously.
The above configuration snippet enables the embedded reports so when embedded reports are created, those reports will not go through the authentication process, these will be authenticated anonymously.
The next set of configuration stanza enables the following
- Introduces the LDAP authentication for the URL context /shc
- Set the X_REMOTE_USER with the user name of the authenticated identity
- Inserts the ROUTEID cookie that will be used for the sticky load balancing, it is critical to make the LB work correctly.
Include the following line as part of httpd.conf
This file has the definition of the backend search heads that participate in the search head clustering. The content of the conf/extra/httpd-LB.conf file looks like this:
Once you restart the proxy with these settings, it will be ready for testing SHC via the load balancer.
How to test
Invoke the proxy url http://proxy.sv.splunk.com:8080/shc, it will redirect to one of the back end server. You can inspect the requests sticking to a particular back end SH, based on the value for ROUTEID. As you can see the screen shot below taken from one of the session via the reverse proxy. The cookie value is computed as SHC.ROUTE as defined in the httpd.conf. If you have noticed the user name in the splunk web nav bar is missing, then all the likely it is due to non sticky load balancing.
Embedded Reports Can be created from teh splunkweb, when you create it will automatically populate the anonymous url with context /gpass as you can see below