Awhile back I posted a video on setting up a Drupal sub-theme.
Drupal core announcements: Drupal 8 theme system critical issues sprint (June 5-7 in Portsmouth, NH)
Join us at the NHDevDays 2 sprint in Portsmouth, New Hampshire June 5-7.
The sprint will be focused on resolving the two remaining theme system criticals around autoescape and SafeMarkup:
- #2280965: [meta] Document or remove every SafeMarkup::set() call
- #2273925: Ensure #markup is XSS escaped in Renderer::doRender()
Thanks to funding from the Drupal Association via Drupal 8 Accelerate to cover travel expenses two of the Drupal 8 theme system maintainers – joelpittet and Cottser – will be in attendance leading the sprint and providing guidance.
We recommend you have Drupal 8 installed before the sprint and if possible have some experience working on Drupal core (attending core contribution mentoring hours would be a great way to get a kick start!).
The theme system maintainers will be giving a brief overview of text sanitization and autoescaping in Drupal 8 and you will learn more about working with the theme layer in Drupal 8.
We also welcome remote attendees, join us in #drupal-twig and RSVP – there is a remote sprinter option when registering.
This is a recap of our session from Drupalcon 2015 in Los Angeles.
About 6 months ago, the Drupal security team announced one of the worst bugs in the history of Drupal. The recovery period has been very challenging for our community, though I think it's safe to say now that the storm has passed.
One of my personal sites, http://whaleocalypse.com/, was affected by the attack. I've been able to verify that the attack I experienced was the same attack that an overwhelming majority of Drupal sites experienced. What follows is a comprehensive post-mortem on that attack, including how it was done, and how you can ward off similar attacks.
A couple of disclaimers before I begin. First, I'm going to confess to some very insecure practices on whaleocalypse.com. Understand that whaleocalypse is a play thing, I knowingly took huge risks with that site because I had nothing at stake if it was lost. Quite honestly, I was pretty excited to learn that this disposable site been hacked because it gave me the opportunity to learn more about security. I would never take these same risks with my professional client work.
Second, and perhaps more importantly, I'm going to reveal how this hack was pulled off in very, very specific detail, including a large amount of fully functional exploit code. Checkout the code on Github. If at any point you feel yourself going "Hey, you're telling people how to hack my site!" I'd invite you to stamp that feeling down. If you are still vulnerable to any of the techniques described here, I'm sorry to say, you were already hacked weeks ago.Table of Contents
- How This Hack Works in the Wild
- How This Hack Works in the Wild: Video Explaination #2
- Why create all these backdoors?
- Was This Really Drupalgeddon?
- How I Recovered
The problem lies in database.inc, specifically in the function expandArguments(). Note lines 739, 745, and 755 in that file. It might be a little hard to follow, but the gist is that Drupal trusts the value of $i on 739 without sanitizing it. This is a problem because, in the case of the user login form, $i is generated based on the name attribute of the html <input> element.
Which means this hack just so, so, so easy to pull off:
Using this vector, I trivially worked up a little drupalgeddon client that will allow me to inject complex SQL statements into any vulnerable site.
Here it is in action:
Checkout Stephan Horst's examples to get a broader sense of what's possible with this bug.How This Hack Works in the Wild
It's all well and good to talk about the theoretical danger presented by this bug. What I want to know is: how was the bug actually used in the real world by the real hackers who broke into my site. In order to answer this question, I spent a great deal of time examining the exploit files left on my server, and reverse engineered the tools that would have been used to create them. Check it out on Github.
I want to note here that, according to data released by Acquia 68% of hacking attempts in the first week of druppalgeddon were of the variety I'm going to describe here. The second most common attack style—adding a new admin user—accounted for 26% of all attacks. The user add style of attack is well understood, so I won't go over it here.
The attack that I, and most drupal site owners, experienced I'll call the "menu_router" attack. It works as follows:
- Create a new entry in the menu_router table whose "access_callback" is file_put_contents and whose "access_arguments" are a filepath and the contents of a file. This creates a new page in drupal to the effect of http://example.com/backdoor, where "backdoor" is a random string.
- Visit the new exploit page http://example.com/backdoor. When Drupal checks to see if the user should have access to the page, it will run file_put_contents('path/to/backdoor.php', '<?php //contents of attack file here ?>'). This creates a backdoor file which allows the attacker to execute any PHP on your server.
- Using the newly created backdoor.php, run some code which will make a request to another server and download a secondary attack file. This secondary attack file is a file upload form. It gives the attacker the ability to upload any file to your server.
Also, here's another video overviewing the whole exploit kit, and the defense kit.
Video table of contents:
- Intro 0:01
- The Anatomy of a Hack 0:55
- Step 1: Inject the SQL 4:17
- Step 2: Create a Backdoor to run PHP code 10:54
- Step 3: Create a Backdoor to Upload Files 13:14
- Defense procedure 1: prevent SQL injection 22:21
- Defense procedure 2: prevent malicious file uploads 29:01
- Defense procedure 3: prevent arbitrary code execution 34:15
Using some strategy akin to the SQL injection tool I showed above, the hacker would have inserted a new row into the menu_router table. That row looked like this:How to Defend Against This: Drop All POST Traffic in Varnish
If you haven't yet, UPGRADE TO DRUPAL 7.32 OR HIGHER. If you are reading this, and you are running a Drupal site on version 7.31 or lower, your site has already been hacked, probably several times by many different individuals.
That said, this is a good opportunity to harden your site against all SQL injection. If your site has a handful of logged in users, and serves mostly anonymous traffic, one precaution you can take is simply rejecting all POST data in Varnish—almost all SQL injection vectors will make use of POST traffic. You can do this trivially in your VCL file like so:
Site administrators can still login by visiting Apache directly on its new port, which is now 8000. You'll prevent non-administrators from visiting Apache directly by using an IP access control list in your Drupal .htaccess file. To accomplish this, simply prepend the following to your htaccess file:
Now, when a hacker tries to POST to your site they'll get an error:
See: Defense Procedure 1 at 22:20 in my video above.Step 2: Create a backdoor that allows you to run any PHP
Now that we've created our menu_router entry, it's time to visit our new page. When we do so, a new file will be created.
As soon as the attacker visits this page, assuming your file system is writable, the following file will be created:
I spent a long, long time staring at this file during my post mortem. It was finally this comment on drupal.org that cracked the case for me. Can you tell what it does? Spoiler alert! It's a backdoor that allows the attacker to run any PHP code. Once this file is installed on your server, all the attacker has to do to execute PHP is send an HTTP request like so:GET /modules/poll/zkwv.php HTTP/1.1 Host: exploited.com Cookie: Kcqf3=base64_decode; Kcqf2=["preg_replace", base64 encoded]; Kcqf1=[any PHP, base64 encoded]
I've reversed-engineered a deobfuscated version of it, to make the inner logic visible:How to Defend Against This: Set Proper File Permissions
Its worth noting here, that the most common attack seen during Drupalgeddon would have failed on any site that had set proper Unix file permissions. If you haven't already, you can do so automatically with this script. See Defense Procedure 2 at 29:00 in my video above.Step 3: Create a backdoor that allows you to upload files
Now the hacker has the ability to run any PHP on your server. And of course there's a great deal of damage that is possible with that power, but I've been able to determine what this hacker actually chose to do with this power. In addition to the arbitrary code execution backdoor I demonstrated above, I found several attempted backdoors. The most telling was this one, which I found inside the locale module:
Note, that is not my 404 page, that is the attacker's 404 page. Which means my server tried to execute some code intended to download a file. Which also means that we can go and see what file the attacker was trying to download:
I recognize this file! All told, I found four copies of this file on my server, and two copies of 404 pages in which the attacker attempted to download this file. When you compile it into HTML it's an uploader page, which will allow the hacker to upload any file to your server:
So not only do I now know what code this attacker executed and what it was for, I know the name of the town in Romania where he or she is fromHow to Defend Against This: Upgrade to PHP 5.5.0
Remember, this is the second of two backdoors the attacker is going to install, and it is wholly dependant on the success of backdoor number 1 (which gave the attacker arbitrary code execution). Backdoor #1 is wholly reliant upon a weakness in preg_replace which allowed you to evaluate the haystack as PHP. This weakness was removed in PHP 5.5.0.
So while the attacker might still get backdoor #1 installed, they won't be able to use it, and they will not be able to create backdoor #2.Putting it All Together
Considered together, you come up with a script like this. This is my best guess at the script this hacker actually used to attack my site. Once a hacker has executed this script against your site, he or she will be able to inject any SQL into your database, run any PHP on your server, and upload any file to your server.Why create all these backdoors?
OK, if I may speculate wildly for a moment: this attacker was not interested in changing any content on my site. They didn't deface the homepage, or use my server for spam as far as I can tell. All they did was install obscure backdoors with random file names. So if I had to guess, the goal of this particular attack was to compile a large database of backdoored sites, and then sell the database. So maybe eventually this could have been used to steal data, but that doesn't seem to have been the direct goal of this particular attacker.Was This Really Drupalgeddon?
In a word, no. A two words, not hardly. In a gif:
Let's just review some facts:
- Anyone hosted on Acquia, Pantheon, Blackmesh, or Commerce Guys Platform.sh was safe, whether or not they patched.
- Anyone who's file system was unwritable to Apache was safe from the most common attack.
- Anyone running php 5.5.0+ was safe from the most common attack.
The discussion of this bug has so far centered largely on it's potential for damage. And, fine, I admit in theory this could have been really bad. But here in the world where we actually live, it's just impossible not to notice—nothing bad really happened. I've read lots of cases of hackers creating fraudulent users and uploading backdoor files, but I have yet to find—despite considerable effort—a single person blogging about a `DELETE FROM node` attack. As I learn more and more about this bug, I am increasingly persuaded that such an attack did not occur.
As Stéphane Corlosquet wrote on the acquia blog, across tens of thousands of (failed) hacking attempts, none were destructive, or even visible to the end user:
We could not find any query intended to change the content or destroy sites: attackers were only interested in installing backdoors to take over the site or server at a later point in time, and make the intrusion unnoticeable.
So just a casual suggestion among friends, but maybe we should stop analogizing this bug with the literal end of humanity and start regarding it as what it was: another day in the life of a sysadmin.How I Recovered
TL;DR I restored from backups. Blair Wadman gives a pretty good overview of what it takes to secure your server, but briefly I:
- Used the drupalgeddon tool to check for exploit files (it found all of them, by the way).
- Copied my whole codebase from the server down onto my desktop and diffed them against the last known good version in git (this is how I know the drupalgeddon tool worked).
- After 360 days and 5 hours of continuous uptime, I re-installed Linux from scratch. It's a good thing too, because I've been meaning to patch the shellshock vulnerabilty (lol).
- Deployed my local backups to the new server.
I just want to close by saying that I think this security event could have been a lot worse. The big Drupal hosting providers were patched before the vulnerability even became public, so very few high value Drupal targets were hit. Of the Drupal sites that did get hit, the most common attack seems to have been pretty innocuous. This is a case where open-source worked more or less as it's supposed to; more eyeballs on the problem brought an incredibly obscure bug to light, and let us fix it—whereas a closed source product would have kept chugging along with the vulnerability unpatched. While this has not shaken my faith in Drupal as a whole, it has made me sit up and take security a little more seriously.References
It’s almost that time of the year, where we get the Drupal community together in Europe to celebrate and share amazing stories. In Front end we are coders, designers, and sometimes both. We build themes, work with developers, designers, project managers, content strategists, marketers and clients. We are looking for a mix of topics that will inspire and excite us.
The Front end track will have 13 sessions and we would like to cover the following topics:
Drupal core announcements: Formal usability testing of Drupal 8 at the University of Minnesota Usability Lab, 22nd–25th June
On the 23rd–25th of June, just before Twin Cities Drupal Camp, we will be conducting usability testing focusing on Drupal 8 at the University of Minnesota. This is a great opportunity to evaluate the current state of Drupal 8 and identify issues that can be resolved before release, or require much of our attention after release.
We'll start on fixing items found during the Twin Cities DrupalCamp Sprints and continue throughout Drupal 8's release cycle.
The University of Minnesota's Usability Services Department has been an amazing long-time supporter of the Drupal project. Hosting us in 2008 just after Drupal 6's release for the first-ever Drupal formal usability study, and again in 2011 just after Drupal 7's release. These usability test results have been invaluable in shaping Drupal's user experience over the years.What will we be testing?
The tasks for this study will varied and focused around both content creation and site building activities:
- Mobile content creation experience
- Content authoring (preview, menus, in-place editing)
- Layout modeling (placing blocks and editing blocks)
- Content modeling (Field UI, Views)
We will be inviting users with a technical background, of which at least half have experience with Drupal 6/7.
We are close to release thus we have a small window of opportunity to fix these problems before Drupal 8 is released.
- Review the test script (should be ready next week, to be finalized by June 15).
- Help find and fix any major user-facing bugs prior to Beta12 (June 17) that will get in the way of users completing the test script. File these under the UMN 2015 prep tag.
- Attend the usability testing, either in-person or remotely (see below).
- Contribute during the Twin Cities DrupalCamp sprint (in-person or remotely in #drupal-usability) to translate problems that were found into actionable Drupal core issues.
- Provide solutions/reviews to the identified issues with the UMN 2015 tag.
While space is limited, we are able to accommodate some community members who wish to attend the usability testing sessions either in-person or remotely (over WebEx). In case you are interested please get in contact with Lewis Nyman.
We know that experiencing a usability are quite transformative, and hope that anyone interested reaches out. Sadly, the sessions will not be fully open - as we wish to respect our participants' privacy. Attendees will be required to sign the University of Minnesota Usability Lab's Code of Conduct in order to ensure the privacy of testing subjects is upheld.
We are very excited about learning more about our users and Drupal, and hope to share the results with you as soon as possible!
One of the most commonly used modules for helping with a Drupal 7 site’s SEO is Metatag. A rewrite of the Drupal 5 & 6 module Nodewords, Metatag provides a flexible system for customizing the meta tags used on a website. Using various meta tags it is possible to exert some control over how search engines analyze each page, and how the content looks when it is shared over social networks.
Yes, yes, one click deployment is amazing. And yet so last year. So I say, hell no, we want one push deployment. The dream cheeky USB button is neat. Let's deploy sites with it. This project is an example of how you might be able to do it: Deploytron.
Ok, so what does this really mean? With virtualized development environments, continous integration tools, and continous delviery tools we should be able to make changes to production sites predictably and easily. Deployment isn't an after thought, it's simply another step in the development process. And its easy. Hosting services are marketing tools to do specifically this. But frankly, it takes the fun out of it.
You should be able to launch a site with some fan fare. Deliver a Deploytron box to your client and let them hit the button. Put names in a hat and let somebody in the office press the button. See how long it takes your cat to step on it.Categories: Planet Drupal
The next beta release for Drupal 8 will be beta 12! (Read more about beta releases.) The beta is scheduled for Wednesday, June 17, 2015.
To ensure a reliable release window for the beta, there will be a Drupal 8 commit freeze from 00:00 to 23:30 UTC on June 17.
Recently, I had the opportunity to present my core conversation,”Pain points of learning and contributing in the Drupal community,” at DrupalCon Los Angeles.
My co-presenter Frédéric G. Marand and I talked about the disconnect between Drupal and api.drupal.org on core and some of the pain points to contributing and learning in the Drupal community. We also spoke little bit on continuous contribution and sporadic contribution and benefits of both.
The open mic discussion brought up some interesting issues, and so I have compiled some links to answer questions.
Audience Suggestions and Responses
- Stable release of Drupal 8 will help people start on client work and support contribution. The Drupal community needs to recognize contribution not just in the form of patch, but mentors mentoring on IRC during core office hours, credit to code reviewers on the issue queue, recognize event organizers and have people edit their profile on Drupal.org and list their mentors at the end of a sprint.
- We now have an issue on Drupal.org to allow crediting for code reviewers (and other non-coders) as first-class contributors.
- Make profiles better on Drupal.org. Here is an issue for that – [Meta] new design for User Profiles.
- Event organizers could get an icon on their profile page. You can read more on that – Make icons for the items in the list of user contributions to be included on user profiles
- Another issue to read – Reduce Novice Contribution differences and consolidate landing pages, content, blocks
- Explanations of what needs to be done could be a big time-saver. For Drupal 8 there are pretty clear outlines of what could be done for core.
- There was a suggestion to provide video and audio documentation instead of just text, walking people through issues. There are four or five companies that make videos and we have core office hours for walking people through the issue.
- A few people expressed that its hard to keep up with IRC and are looking for easier ways to communicate. I have created an issue for that and you can read more here – Evaluate whether to replace Drupal IRC channels with another communication medium.
- Another audience member suggested that we need to make sure that communications that happen in IRC are summarized and documented on issues so more people can get familiar with the discussion.
- There were some suggestions for core mentoring that have been proposed but haven’t panned out such as twitter or hangouts (privacy concerns, less office-friendly)
- Someone suggested that those who don’t like to get on IRC, can get core updates via email (This week in Drupal Core) which is a weekly-to-monthly update on all the cool happenings in Drupal 8
- Users can also subscribe to issue notifications in email on the issues/components they want to follow on Drupal.org.
Overall it was an enlightening core conversation and it was amazing to hear from the community about their pain points and suggestions they made.
Jeff talks to Deane Barker of Blend Interactive about the art and practice of content management, the joy of solving complicated problems, and his upcoming O'Reilly book Web Content Management.
Before anything else, I want to point out that while this post will focus on maintaining a contrib module on drupal.org, that is just one of the many ways to contribute to the Drupal project. Every contribution is important whether it's a core patch, a documentation edit, a translation, or something else. If you use Drupal, please consider how you might be able to give back to the community. If you're already contributing, then thank you!
In April 2015, I was excited to talk with John Doyle, General Manager Technology & Solutions Architecture at the Australian full-service digital agency Komosion, to explore their decision to adopt Drupal to replace other technologies, including an in-house CMS they'd invested 10 years of work in. In this podcast, John very clearly lays out what Komosion's priorities were in making this decision, the benefits for the agency and its clients, and the future he sees using Drupal as the basis for future work.
It came up multiple times at recent events that it would be very helpful for people significantly working on Drupal 8 critical issues to get together more often to talk about the issues and unblock each other on things where discussion is needed. While these do not by any means replace the issue queue discussions (much like in-person meetings at events are not), they do help to unblock things much more quickly. We also don't believe that the number of or the concrete people working on critical issues should be limited, so we did not want to keep the discussions closed. Here is the recording of the first meeting from today in the hope that it helps more than just those who were on the meeting:
Unfortunately not all people invited made it this time. If you also have significant time to work on critical issues in Drupal 8 and we did not include you, let me know as soon as possible.
Last week, some colleagues from Cocomore and I attended DrupalCamp Spain 2015. Spanish Drupal community is awesome, and they have put all their efforts in making an unforgettable event again in this 6th edition (the 5th I have attended).
The event was divided into different activities for the three days: Business Day and Sprints on Friday, and sessions on Saturday and Sunday.
On Wednesday’s blog post, we debuted our first interview from DrupalCon. If you missed the Roundtable Interview, you can catch the first episode here. The interviews first appeared during our weeklong live broadcast with Periscope and Twitter.... Read more
Barcelona is one of the most cosmopolitan cities in Europe and we want to infer the same spirit for the Coding and Development track. This is an exciting moment in the web development industry and we plan to take advantage of that. The DrupalCon Barcelona Coding and Development track is focused on preparing developers for the future of the web development universe.
Any morning starting with DrupalTour journey is great ;) Especially if the point of destination is only 70km away. This time we visited Rivne, welcome on board!
We decided to slightly change the concept of the event. It’s much more convenient to communicate with visitors in calm atmosphere of conference halls — we all can concentrate purely on reports and questions.Read more
In the past three posts we've looked at how the Charlottesville Expense Budget might be made more transparent through maps and visualizations. The final stage in our process is getting the data we've collected added to a larger data set - in this case we're going to use openspending.org as a repository for our work. This means that even if this site fails (gasp) the work we do will not be lost, and may contribute to a larger and greater good (perhaps a stretch right now, whatev).
Using views data export we can create a CSV file that has all the needed fields for openspending.org - they're a pretty flexible group Amount and Date are the only required fields, we've added Merchant (thanks to the SCC data for that) as well as categories and locations (thanks to John Pfaltz for that).
A problem we hear about quite often is a huge cache_form table. Drupal by default caches all form data (typically for 6 hours). During peak traffic times, that can be a lot of form data being cached. When the form cache is cleared during Drupal's cron, the following DELETE query is used:DELETE FROM cache_form WHERE (expire <> 0) AND (expire < UNIX_TIMESTAMP(NOW()));
However, InnoDB does not release disk space back to the file system on DELETE queries, and that can easily lead to a very large .ibd file for that table.
The three most common "fixes" we see being discussed are:
- mysqldump - Many people recommend exporting the table via mysqldump and then importing it again. While this does reclaim the disk space, your cache_form table will be locked during the dump and the import, and any users who filled out forms during that small time gap in between the dump and the import will have their form data lost.
- OPTIMIZE TABLE (or ALTER TABLE "Engine=InnoDB") - Another recommendation often seen is to run OPTIMIZE TABLE on the cache_form table. However, this will also negatively impact your users by locking the cache_form table during the process. The ALTER TABLE is also mentioned here because it essentially does the same thing on InnoDB.
- TRUNCATE - Probably the most common recommendation we see is to simply TRUNCATE the cache_form table. That will instantly free up all of the disk space that table is using, but it will also delete ALL cached form data.
Fortunately, Percona Toolkit contains a great tool to do online schema changes, and it even works if you are running a cluster. We have a client whose cache_form table legitimately grows to ~20GB during peak traffic times, but their site has almost no form usage outside of normal business hours. Running the following as a maintenance task just before the start of the day when there is the least amount of data that legitimately resides in cache_form typically results in a shrinkage down to less than 50MB:pt-online-schema-change --host 127.0.0.1 --port 3306 --user bender --alter "ENGINE=InnoDB" D=database,t=cache_form --execute
This runs an ALTER TABLE, but with a few extra things going on under the hood. Instead of simply copying the table, it also creates triggers on the original table so that any new data being written to the original table are also written to the new table. When the process is done, it then atomically renames the tables and drops the original.