Web Server Sessions or: How I Learned to Stop Worrying and Love Unplanned Outages
Posted on June 21st, 2014
Hello. My name is Joseph Tole but you can just call me Joe and I am the new CTO for 7L Networks. I’m pulling a late night at the data center but, anyone who has ever operated in a data center or enterprise class production systems knows that sometimes, late nights are a requirement and it comes with the job. Also, in case it’s not obvious from the title, I’m a bit of a Stanley Kubrick fan (see Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb). Coincidentally we both have the same birthday.
I wanted to take a break from my work and write my first blog post. This post is typically aimed at developers and systems administrators but I’m hoping that anyone in IT can appreciate it.
I wanted to talk about sessions from the web developers point of view. Sessions, in this case, are referring to when a web based application needs to remember something about the current user, for example, when they log into a web page and the server side code needs to remember they are logged in so it can show them content relevant to their account. Sessions are obviously very important in web design because without sessions, pages would have a hard time keeping track of important data from other pages or being able to direct specific data to specific clients. Many web servers make it easy to create and use sessions and while this is often very helpful for entry level application design as it simplifies session management from the programmers perspective, it starts to create complications as a web site needs to expand, scale-out or provide high availability.
The problem with web server managed sessions is that they are local to the web server. When your site is only on one web server then this isn’t a noticeable concern but, once you’re page needs to be on more then one web server, this starts to cause a problem. When pages are replicated on to multiple web servers, it’s common to use a load balancer to balance page queries between different web servers but this is where the problem happens. Suppose a user logs into a web page where a load balancer directs them to the server www-1 (the first web server). They fill in their login credentials and click the login button where they are then directed to the page for a logged in user. As they connect to the page, the load balancer sends them to the server www-2 (the second web server). If the session is being handled by the web server then www-1 will know that the user is logged on but www-2 won’t have ever seen the user before and the user may be asked to login again or may be given a access denied message or many other possible outcomes. The problem with sessions localized to a web server can and, typically will, extend to far more issues then just the login page itself but the login page is a good example of what can happen when you localize the session to the web server.
Most load balancers have a way around this problem which they call sticky sessions. An example of sticky sessions would be if the load balancer hashes the source IP address and use the hash as a token to ensure they always reach the same web server. Another method I have found is that the load balancer may inject a cookie indicating which web server the initial connection was directed to if it doesn’t see that cookie already in the cookie headers. If it does see that cookie then it will use the value to direct the client to the same server. The cookie method allows for balancing methods like round robin, least connection servers, least transfered data servers etc. While those methods will ensure that the connections between backend web servers is more evenly distributed vs. the client source address based method, these are still sticky sessions with long life times to the same server over many connections. This can cause a skew in server traffic balancing over time where some web servers may have a much higher traffic load then other web servers.
While sticky servers may seem like a good solution to allowing you to load balance data between servers without having to stop using web server based sessions, they’re still just a crutch to the bigger problem. Suppose there is a security vulnerability on your web servers and in order to patch this vulnerability, you need to reboot your web servers. Well if you have load balancers then this usually isn’t a problem as most load balancers will allow a gracious stop of traffic to any web server where they will no longer send new connections to that web server and will allow you to know as soon as all existing connections are finished so you can shut down a web server without interrupting traffic. All new connections would be directed at the web servers that are up so that if someone is receiving data from a web server you plan to shut down, they will continue receiving that data and then when they click on another link / page, they will be directed to one of the other web servers that are not being shut down. This method would allow you to perform a upgrade cycle on all web servers where no single web server is down at the same time so that all web servers can be patched and rebooted separately without causing any outages seen by the clients. Another benefit to the load balancer is that if a web server crashes, the load balancer can redirect them to another web server which hasn’t crashed and the user would only receive a error on a page they are receiving as the server crashes which would then be sent to a new server as soon as they hit refresh.
This is where the crutch aspect of sticky sessions comes into play. If a web server experiences either a planned outage or a unplanned outage, either one, the user will be directed from the server which they were being bound to which knows who they are via the sessions to another web server which isn’t aware of them and would ask them to log in again. Again, this can be more then just being asked to log in again and important work may be lost.
So now that I have covered all of the flaws about why web server based sessions are bad, you may be wondering what the solution is. The best approach is to migrate your sessions to databases. Ideally everything about a users session should be saved to the database you use. When a user logs in, they should be issued a cookie representing a session ID. This would be a ephemeral key that would change on each login. Every page that you write can use the session ID to query the database and get information about that user. For example, when the user John-Doe@example.net logs in, they can receive the cookie sessionID with the value a43f2e1d-c66c-484a-a477-be9cfffa7bc7 (UUID’s are a good practice for session ID’s). Every web page they visit can then query the database for the session ID and see that user is John Doe, he logged in a minute ago, his log in IP, his email, etc, regardless of which backend web server they are directed to.
By relying on the session ID and querying the database for the details, this means that the web servers do not have any localized data about the user and if a user is constantly changing which backend web server they connect to via the load balancer, they will always be treated as the same logged in user with the same accessible data, regardless of which web server they connect to. Obviously a users session ID would need to be protected via SSL/TLS so that someone cannot eavesdrop on the key and impersonate the user. If this is done right then there is no need to provide sticky sessions to users, no loss of content or connection properties between backend servers, short lived sessions on each server allows for a far more balanced load balancing, etc.
Another key benefit to pushing the session ID to the database server instead of the web server is that once a user logs in, they can remain logged in for as long as you would like them to without having to be concerned about the load it would create on the web server by requiring a long session timeout. It’s usually problematic to have long session timeouts when a web server is responsible for the session length but if the session data is stored in the clients machine via a cookie and accessible to your web application via a database lookup then there is no overhead if you want a users session to remain valid for a week, a month or a year without being prompted to log in again.
There are many web frameworks which can automate using a database for session management. This is a built in feature of my favorite framework, Django, but other frameworks also support this feature. If you need to speed to up the query time to the database server or reduce the connections to the database server for session data then a database caching framework like memcached can greatly improve both the query time and the database connections.
I hope this blog has helped explain why relying on web server based session management can be problematic and even more importantly I hope you enjoyed reading it. Feel free to share your comments with me as I would be happy to read them.
— Joe