Why (not) to use Grsec / security in your system.
That said, on with the post:
What is Grsec?
Grsec is a kernel modification, designed to help protect your system from hacking, etc. Many times, it does just that, although, many times, it has been known to cause problems.
Design:
There's a reason that most kernel modifications (audio, eth0, etc) are modular, rather than compiling them into the core itself. Each and every bit that you compile into the core kernel will cause just that much more load on your system itself when things are being processed. Is this good? Well, is high load on your system good? No, it's not. I'm not saying that grsec is always the cause of high load, but adding that much more to your system core doesn't HELP matters any, now does it?
The difference between modular design and straight forward compiling something into your kernel is with modular design, it's easily removed.Typing insmod or rmmod (with the correct arguments) will usually do the trick.
Speaking of removal: Ever tried removing grsec? It's not as easy as some might say. I've compiled and installed rpm kernels quite a few times in my day, and it's not as simple as "updating" the kernel (or loading a new one). Every time I tried this very simple approach to grsec kernels, it's resulted in catastrophic downtime, usually resulting in dc being called to boot into the grsec kernel again.
Why would you want something in your system that basically takes control of your kernel, the core of any OS, any system (mac, windows, you name it), and certainly isn't that easy to remove? I certainly wouldn't.
Grsec is designed to LOG most actions, flooding an already overworked syslog, causing more hard drive seeks, again, causing more system load on the server itself, and causing more and more individuals to flat out ignore system logs or messages because they're "too big". Well, that can only spell Trouble, and with a capital T. As soon as you start ignoring those system logs, forgetting to check them, you open up doors for x and y hacker to come in.
Security:
Security is best handled in layers. Most, with grsec installed think "Oh, I'm secure". Well, you're not, at least not with grsec alone installed. In fact, if you're connected to the internet, or even turned on, you're not secure. Am I kidding? No, I'm not. There's ALWAYS going to be some kiddie that'll get around something, ANYTHING. The trick is to catch 'em before they do too much damage, and to listen to what your system is telling you, NOT to install something like this and assume it's done.
Stability:
Over the years I've seen various actions on various systems caused by grsec. A couple of examples follow, and you can tell for yourself just how dangerous trusting your system to grsec is.
Client A:
I've done game development for a good few years, never had an issue with doing code work for various people I know, usually at very low cost (if any). One of my recent projects just happened to be on a server with the exact same stats (cpu, ram, etc). In fact, my server at the time was more overloaded, due to phpmonitor and sql tying it up to no end. In fact, the only major difference between THEIR server and mine was the kernel (they were running grsec, I wasn't).
when running on their server, the game would dump every few hours, literally just hang, until one of the admins came up and looked at it. When running on mine? It would run as smooth as anything, no issues whatsoever. When running on the server before their current one, again, it would run no problems, whatsoever.
Coincidence? I thought so, untill I looked at the core, found out that the game itself was hanging on a specific call. I fixed the call (was a system() call), modified it to work with grsec, but that shouldn't have had to be done in the first place
.Client B:
One of my regular customers contacted me a while back regarding server load. As usual, I logged in, found (most of) the problem, solved it, and logged back out. When I logged in, the load was between 2-3. When I logged back out, the load was settling just under 1. Unfortunately, no prodding, pushing or pulling could get it to go any lower than that. Well, I suggested the client dump grsec and see what happened. The system had been hanging left and right in the middle of the night, etc, nobody knew why. In fact, the very next morning, the client called me, and it turned out that apache had frozen (wouldn't restart, wouldn't do anything). I rebooted, the server, came back up just fine. Still was loading right around 1, constantly though, so again, I suggested removal of grsec kernel, and that he contact his dc to do this. He said he would, and when they did it, to this day, I've yet to see that load go above 1.
Coincidence? Not when you consider that the system is a dual processor system with 2g ram. Something definitely was wrong there, and it looks like removing grsec brought it back to normal.
Client C:
Last year sometime, I started working for a client who had some pretty overused webservers. One of these was constantly crashing (every day), and when it came back up, I'd go through the messages, only to find that grsec itself had caused the crash. The cause? Memory overrun somewhere in apache. Every time!!! Had the DC remove it, and guess what? No more crashes!!!
Client D:
Just after I started getting into running LTN, I was contacted by a client who had some system load issues, bad partner, bad things all around. I did my usual setup, and whatnot, only to find that certain system commands weren't available because of a "grsec" kernel. Rather than remember root passwords for every client, at the time, I had a default user on systems (still do for emergency purposes) with sudo access. Well, on grsec systems, at the time (probably still today), sudo was broken. I mean, it flat out wouldn't work. Followed instructions, still didn't work. Tried again, still , no dice. Eventually, I just gave up and used the root password, but that somewhat defeats the system I had in place.
System security is good, in fact, it's GREAT, but one thing you need to keep in mind is actual usability of the system. Remember, you have users that need stuff too now. For example, how would you feel if your host suddenly said "No, I'm sorry, mail is disabled in php", out of the blue. Would you thank them for that? No, most would get up and leave, because mail is one of the most used functions IN php. In the same respect, would you stay with a system where load was constantly high enough to cause the system to be slow and unresponsive? Most wouldn't, yet grsec adds TO that system load, instead of detracting from it. Not just once in a while but on a constant, regular basis, as it is part of the system core itself.
There are plenty of ways to secure your system, grsec is JUST one of them, and, honestly, the most consuming, problematic one that I've seen to date. Like I said, remember, you DO have users on this system that need the functionality that's taken away by grsec, in many many cases.
Should you trust your system kernel to an application made by someone who has no place in the kernel development? I certainly don't, and wouldn't advise it to be done.
Should you trust your system kernel to an application that requires being compiled in, rather than being modular? I certainly don't, and most definitely wouldn't advise it to be done.
Should you modify your system kernel period without knowing 100% of what you're doing? Again, I wouldn't, and I don't advise it being done. Remember, this is the core of your system you're talking about, the heart and soul. If you screw one thing up, your system could quite possibly be unrecoverable, and then what?
Like I said before, I have nothing against the grsec team, or grsec itself. It's a GREAT idea, albeit entirely bloated, and not really designed the best, but, in this case, I've seen entirely too much negative to say it's worth the risk of screwing up the server(s) that do hosting and constantly rely on stability and uptime.
