Below is a story of one of the sites that we support being hacked, and the lessons we learnt from the experience. I hope you will be able to learn from our experience, and make the right decisions to prevent such accidents from happening to your sites.
Two months ago we received an email from the site’s hosting provider, stating that the site “has been hacked and used to run illegal software on the server”, for which reason they’ve completely shutdown both the site and its control panel, until we tell them that we’ve made all the necessary preparations to correct the problem — which meant removing the vulnerability that was exploited, as well as taking steps to reduce the chance of any other vulnerabilities.
I need to note that on the site itself, prior to it being shutdown, there was nothing to indicate any problems. All looked fine.
The way the hack was done was by using an insecure request processing in one of the modules used with the Joomla (a Content Management System) on the site. The module, developed by a third party, was not checking one of the possible request types, which could come from any visitor’s browser, for possibility of malicious code. This meant, that some commands could be executed on the server of the hosting by calling the site in question with the following link:
SITE_ADDRESS/index.html?option=MODULE_NAME&controller=../../../../../../../proc/self/environ%00 HTTP/1.1″ 200 3573 “-” “Mozilla/5.0 mimot
This request should work in the following way — the controller code for the specified module is called, and it is passed the parameters in the call, which get processed by “good” code afterwards. What happened in reality is that the parameters in the request link were not “sanitized” by the code of the module (this is a standard safety procedure, which was left out by the creator of the module by mistake), thus allowing the parameters to contain some “bad” code, which was executed as part of the controller call, before the “good” code was invoked.
The passed parameter contained “bad” code which executed the following actions:
- Moved many folders up the file structure from where the actual site is located (so further code was executed in the environment that is not even available to the site administrator accessing the site with standard means — FTP or control panel). It is possible that the number of levels that the code needed to be places is standard for the hosting server the site is located on, and that folder could contain service utilities used by many other sites.
- Got a remote file “FILE_NAME.jpg” and stored it locally. We checked what this file was — if you looked at it using a text editor, you’d see that it was not an image, in spite of the extension, instead it is a collection of executable scripts that allow remotely performing any actions on the server where this file is installed. Powerful badness.
- Renamed the file to “count.php” (supposedly choosing a file that executes often), thus turning a fake image into an executable file.
- Listed all the files in that directory with a lot of additional information (perhaps to use for further attacks).
Upon receiving the notification from the hosting provider, with the above link information, we took the following vital steps:
- Don’t panic. It doesn’t help anyway, but also we concluded that the site is now closed (until we remove the vulnerability), and the malicious code was already removed by the hosting provider, so no more harm can be done.
- Study the threat. We never saw anything like this before, so it took some time to understand what the above link was doing and how, and what needed to be done to prevent it.
- Look for other survivors. We searched the forums of Joomla webmasters to see if there were other victims of the same vulnerability — and it did turn out that this vulnerability became apparent just a few days before it was exploited on our site, and the very helpful Joomla community did already provide ways to close the door of the risky module.
- Restore the site from the backup. This is also a very important step, since we didn’t know what other actions the hacker may have done on the site after gaining access to it. It is possible that the database and administrator user passwords were stolen, that additional dangerous code was installed, etc. Thankfully, our trusty provider has an automatic daily backup procedure (which keeps backups of the site, both the files and the database, for 7 days), thus the site was restored to the state before the hacker attack. It was unfortunate that the content placed on the site after this point was lost, but it was an acceptable cost of knowing that the site is now completely clean of any changes by the intruder.
- Remove the vulnerability. Applying the knowledge of the nature of the threat, and the advice from the forum folk, we edited the module code to include sanitizing of parameters processed by the controller code.
- Change the passwords. Restoring the site from backup ensured that it is “clean”, yet the hacker could have obtained various passwords (most notably, the database password, since any CMS needs to store it separately to enable its own code to access the database). So we changed all the important passwords.
After performing these steps, we notified the hosting provider, and the site was enabled again.
One of the things that we also learnt while investigating the threat is that the module used on the site has not been updated by its author for over a year, so it looked risky to continue using it as opposed to something that has more active support and community. For this reason we looked for an alternative that would satisfy the site’s goals and, thankfully, were able to rather quickly find a replacement. Open source community is a great crowd to interact with, and if there is a problem that many people need to deal with, somebody out there is most likely working on a good solution for that problem.
The lessons we learnt from this accident:
- Nobody is invincible. Even if you write all of your code — you may still make mistakes. But if you depend on software or modules created by somebody else – the chances are even higher. That doesn’t mean that we should panic (because — we shouldn’t), instead, we need to do everything possible that the risks are minimized, and that there is a plan in place for dealing with problems before they arise.
- Backups. Without these, we’d have a lot more work to do. So, when creating a site, always think about periodic backups, whether they are done by the hosting provider, or whether the site backups itself on a cron job. These are vital, and we were truly thankful we had one.
- Keep the software up to date. Since any site usually depends on several third-party modules, and vulnerabilities may often be discovered and fixed in those modules, it is important to make sure that you are always running the latest and most secure version of the software. I love how Drupal CMS deals with it — there is an “Update” module which periodically checks for any third party module updates, and notifies the site administration if an update is needed. For Joomla such a centralized solution does not seem to exist, so we put in place periodical reminders to check whether there are any updates to the components, modules and plugins used on the sites that we maintain.
- Minimize the risks. This is a general statement, and needs to be applied to the specific site configuration. At least one can say that it is best not to use any third party addons unless really necessary, and, if needed, then you should choose modules that are well maintained, have an active community around them, and solve as many tasks of your site as possible – so that you don’t need to rely on too many of them. There are also many security checklists which describe how file access etc should be configured — these should be followed by definition, at least the official one: http://docs.joomla.org/Security_Checklist_7.
- Follow the vulnerability news. For Joomla, there is a security news feed for the vulnerabilities found in the Joomla core itself, and a feed with constant updates on discovered vulnerabilities in Joomla extensions. We are now subscribed to both of those feeds, watching them closely for news about any features that are used on our sites.
And, above all, don’t panic! We hope this information will help you avoid such accidents in the future and, if they do happen, to be able to solve them with least damage to you and your customers.