If you have been dealing with Domino clustering for any amount of time, you undoubtedly know it is a wonderful and frustrating technology all at the same time. For your client users they experience near seamless failover between clustered servers with little or no intervention on their own part. Your web users, however, tell a much different story. Unless you have already addressed this, they have to be the dynamic component of the equation. A web only user needs to know all of the server addresses for the servers containing their email should one or more prove to be unavailable. The method of failover sometimes being merely trial and error. This can be frustrating, and more people are looking for the clustered experience to be feature comparable between the Notes client users and iNotes users. On average, I have been asked about once a week since Lotusphere about making iNotes highly available for numerous customers. I decided to capture one of the methods here on my blog so I can provide the answer completely and quickly to everyone who needs.
-Architectural Discussion-
I want to start by saying there are numerous ways to accomplish this task. The list includes:
-
The Domino ICM – While this is not necessarily a bad option, it requires licensing of the ICM instance if you were to run the ICM outside of a cluster. To ensure your point of entry does not become a single point of failure, and negate the efforts of having the cluster in the first place, you would possibly deploy two servers in this capacity. Another option would be to deploy the ICM on a server within the cluster, but if that were the server to fail, then that would not be very effective. Lastly, you could deploy the ICM on all servers in the cluster, but the overhead and redirection between servers within the cluster could generate unnecessary overhead compared to some of the more elegant solutions.
-
Appliance Based Reverse Proxy – If you have a reverse proxy device in your environment, this would be a wonderful option. You can employ the services of an F5 or Cisco device that is produced by network centric companies, or you could look at solutions like the Whale reverse proxy offering which is application centric with specific rules for know attacks on common web-based applications. These solutions can be extremely robust, with large capacity and the ability to stack or cluster the individual devices to provide amazing levels of availability and service speeds. This all comes at a price, however, and may be a budget buster. Especially if you would have to acquire the appliance(s) if there were no incumbent.
-
Software Based Reverse Proxy – IBM provides customers with a reverse proxy solution, included in many WebSphere and some Lotus licenses. The Edge Component Server has configurations optimized for caching operations or other configurations making setup a bit faster than some of the other solutions. Because it is software, you can leverage virtualization technologies to quickly deploy multiple instances of the solution. Other software reverse proxy solutions are out there, and I am sure virtual appliances as well, but one of the most flexible and versatile could arguably be Apache. Besides the fact that the Edge server is built on Apache, there is a wealth of documentation for Apache, and the cost is nearly nothing. This will be the approach we will examine in this article.
-Preparation-
So now that we have chosen a proxy technology, lets review the players in the solution. Domino will be clustered, which means you will have a pair (or more) of Domino Enterprise servers. This example will make use of Domino 8.5.1 FP1. The server platform is really of no importance since none of the solution is operating system specific. In the interest of simplicity, we will run Domino on Windows 2003. Since Apache will be our vehicle for proxy services, we have numerous options. We could download and install Apache on another Windows server, or we can use a Linux system. There are many quality Linux distributions that are community maintained and are still offered at no cost. I prefer Ubuntu or CentOS. For this example we will use Ubuntu Server 10.04 LTS. If you need help with installing Ubuntu, or want to see how I installed it, refer to this article.
As with any initiative, the foundry work is important to have right. We will assume the following is already in place:
-
Working Domino Mail Cluster – Test that failover, mail deliver, and cluster replication is all performing as expected before moving any farther.
-
iNotes Enabled/Configured on All Cluster Members – Make sure the experience on the cluster members is the same for iNotes users. This includes the session authentication, login page, etc.
-
WebSSO is Working Between Servers – You should be able to freely change between servers without being prompted for credentials if a valid session exists.
-Installing the Proxy-
I started by installing Ubuntu, taking as many default options as possible. Where your particular install, especially for production usage, should vary is in the disk configuration. I did not install the LAMP package option during installation. Since we want to use this system as a reverse proxy, the only piece of the LAMP architecture we would be interested in would be Apache. After booting the system for the first time, we had a few housekeeping items to tend to.
First I installed Apache2 using the apache2 package.
I then proceeded to also install apache2-doc and apache2-suexec-custom per the recommendation from Apt.
Next, I needed to change the network configuration from DHCP to a static IP. If you need assistance with changing to a static IP address, you can refer to the reference section of this guide.
The Ubuntu Apache package makes use of a series of config files for each module you can/would potentially use. They are located in /etc/apache2/mods-available. To enable a given module, you simply link the appropriate config file to the mods-enabled directory at the same level as mods-available. Some mods have more than one file. A .load file is used to load the module at run time, and a .conf file is used to configure that module. We will need to enable proxy, proxy_http, proxy_balancer, and rewrite. There are both conf and load files for proxy, but the others have only load files. Create links in the /etc/apache2/mods-enabled directory for each of the required files using the ln command.
The default settings in proxy.conf are already geared towards using Apache in the way we intend. ProxyRequests is set to off, and the ProxyVia option is enabled already for us. All we need to change is to allow connections through the proxy. Comment out lines 9 and 10, then add the lines as shown in the image below.
Just like the module conf files, there is a default conf file in /etc/apache2/sites-enabled that I removed so no local content displays. Apache also has the ability to read custom conf files after all other configuration files through the conf.d directory. I created an inotes.conf file in /etc/apache2/conf.d to hold the rest of the configuration specific to iNotes. Here is what is in my inotes.conf:
##############################
# Created by: David Greenstein
# Updated: 24MAY2010
#
# Configuration File to Use Ubuntu LTS 10.04 as iNotes
# Reverse Proxy.
##############################
# Replace All URLs to match your iNotes servers.
# EX: domino.collabrisoft.com > myserver.company.org
# Define the ProxyPass Directives
ProxyPass /balancer-manager !
ProxyPass / balancer://inotescluster/ stickysession=JSESSIONID nofailover=On
ProxyPassReverse / http://domino.collabrisoft.com/
ProxyPassReverse / http://domino2.collabrisoft.com/
<Proxy balancer://inotescluster>
BalancerMember http://domino.collabrisoft.com route=srv1 loadfactor=50
BalancerMember http://domino2.collabrisoft.com route=srv2 loadfactor=50
ProxySet lbmethod=bytraffic
</Proxy>
You can download my inotes.conf file here. Modify the conf file to the names of your Domino servers and restart apache.
**UPDATE**
See Chris@GROUP’s blog if you encounter the session persistence issue. You will have to also add in MOD_HEADERS, specify the new cookie to dynamically build to track which back end server is in use for the session, and define the cookie as the stickysession above. This can all be added to the inotes.conf or placed in a new conf file.
With the reverse proxy out of the way, we now need to configure iNotes to work with the reverse proxy. Open the iNotes Web Access Redirect database and setup the database. On the first tab, set the redirection type to Fixed and enter your reverse proxy as the server to use.
The rest of the defaults are fine, unless you need to make adjustments. Save and Close.
I also have configured the DOMCFG.nsf to use the iNotes Login Form for server authentication. Verify the settings in both databases have replicated to the secondary server as well. Restart HTTP or your domino servers.
-Reference-
http://www.ubuntugeek.com/ubuntu-networking-configuration-using-command-line.html
Hi,
ReplyDeleteFor enabling apache modules, you can also use 'a2enmod' command :
sudo a2enmod proxy proxy_http proxy_balancer rewrite
Thanks for the input. Would love to give credit!
ReplyDeleteSomeone had pointed out that the link to the conf file had broken. Apparently the permissions in Google docs changed when I started using an iPhone app with GDocs. I corrected that, but also put the text from the conf file in the post so that does not happen again. Thanks for bringing it to my attention!
ReplyDeleteChris @ GROUP was working on a solution with this architecture and ran into some issues. In his scenario, the session persistence was not staying long enough to allow some of the attachments to upload properly. The session persistence ultimately did not like trusting Domino's cookie. To work around this, Chris loaded the header module (in addition to the required ones for the solution) and forced Apache to set it's own sticky session cookie with the name of the back-end host dynamically set. He then set the new cookie as the sticky session cookie in the balancer configuration, and all was good.
ReplyDeleteI have multiple mail servers reverse proxied by a single apache server. I set a cookie in the browser using a customization in the redirection database to set the right mail server for the apache server to proxy transparently. I'd being willing to share but haven't had time to set up a blog. Would seem like a common dilemna.
ReplyDeleteHello Keith, please could you share your work on my email id.. I can publish it on my blog with your reference. This could be really helpful for the community. My id is tiwari.vikas@gmail.com
DeleteKeith... I am sure many would be interested. You are right that this does seem to be a common issue. If you want to send me your customized iwaredir.nsf, I will add it to this post (with credit of course) and make it available for download if you want to share. Thanks for the input!
ReplyDeleteAbsolutely. I edit the AutoLogin form of iwaredir.nsf and place the following in the $$HTMLhead Field.
ReplyDeletelwrwmrMailServer:=@LowerCase(wmrMailServer);
@SetHTTPHeader("Set-Cookie"; "MailServer=" + lwrwmrMailServer);
Then in my httpd.conf I rewrite/reverse proxy do the following for each mail server in my domain.
ProxyPassReverse / http://mailserver1.domain.com
RewriteCond %{HTTP_COOKIE} MailServer=mailservername
RewriteRule ^/((mail|iNotes|icons|domjava)/.*)$ http://mailserver1.babcockpower.com/$1 [P]
The redirect template already does the work of figuring out what your mail server is so the first snip just sets that value to the cookie. The second snip tells apache which Inotes server to reverse proxy you to.
Any questions, I can be reached at k3ithtaylor@gmail.com
For queries to be sent to the server of the current session is the position stickysession but it does not work.
ReplyDeleteIs that a solution?
@fdrogueres - You are correct in the purpose of stickysession. Read through Chris's post found here: http://www.bleedyellow.com/blogs/lotusnut/entry/apacheinotes?lang=en
ReplyDeleteHe offered a solution to that very issue that works.
@D.Greenstein - Thank you, I also tested this config but it does not work completely. For example if I update to the mail came once in two I change server. So I may have deleted emails on a server that are not yet replicated to the second ...
ReplyDeleteUP !
ReplyDeleteHow to make all requests (ajax, auto-refresh mail, ...) use ROUTEID?
My problem is that the mail refresh does not use the ROUTEID
@fdrogueres: Have you tried Keith's suggested path to correct the issue? I understand the issue you are explaining. When refreshing, because of the proxy settings in iNotes, it goes back through the proxy and you are finding clients switch when this happens. Do you mean use the refresh view icon or the browser refresh button? There is nothing we can do about the users pressing the browser refresh. I would suggest you examine the health of your cluster to see if replication is just to fast enough to accommodate the behavior you are trying to achieve. If you want, email me and we can investigate deeper. dgreenstein at nouveon dot com
ReplyDelete@D.Greenstein : Keith's suggested path is redirect by MailServer, but so I understand that solution don't have failover. In my problem, I Do mean use the refresh view icon.
ReplyDelete@fdrogueres: I would suggest two things to look at. The first is to review the timeout settings for your reverse proxy. http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass Review the documentation here and look for the timeout parameter in the table of options. It explains use/syntax/etc. There is also a reuse flag that you may want to set appropriately so refreshed browsers reuse the previous path to/from the back end. I would also review the cluster its self as I suggested previously. The cluster should be fast enough in a load balanced situation that these messages are deleted in real time. I would look at the workqueuedepth and secondsonqueue stats for both sides of the cluster and ensure they are both below 10 at least. If not, you will see the delays you describe. HTH
ReplyDelete