Malicious page redirects and Drupal's Filtered-HTML text format

One of the Drupal 7 sites I maintain is www.powerpoetry.org which is a platform for publishing poems and comments about them.

Recently, we heard from an irate poet. When browsing to one of her poems, the page would partially load, then automatically redirect to a movie streaming site that had nothing to do with her poem nor with the site in general. Her poem page somehow got hijacked, and readers could never read her work.

It turned out that all site users, even Anonymous ones, had permissions for the Full HTML Text Format. This allowed comments to be submitted that contained malicious code to redirect the page.

So far, I have found two different techniques that were used to implement redirection.

(1)  Use of the <meta> tag with an attribute of http-equiv="refresh". For example,

  <meta http-equiv="refresh" 
        content="2;url=http://scumbags.com/">

where the value of 2 means a delay of two seconds before the refresh.

(2)  Use of the onmouseover attribute to execute JavaScript. For example,

<a href="//tinyurl.com/nm8ugh" 
   onmouseover="document.location='//tinyurl.com/nm8ugh';">

As documented, this attribute and closely related ones are valid for almost all HTML tags, which would make it potentially a big problem where markup is allowed.

If you have good knowledge of HTML, these techniques may be obvious, but for me as a back-end developer they were new.

Because I have direct access to the Drupal database, I was able to use a SQL query such as the following to find malicious comments.

  select entity_id, comment_body_value
    from field_data_comment_body
    where comment_body_value like "%onmouseover%";

 - - - - -

To help plug these security holes, we decided to allow only Administrators to have access to all Text Formats. Depending on the needs of your site, you might want to disable the Full HTML format entirely.

For all other users, the Filtered HTML Text Format was permitted, configured using Limit Allowed HTML Tags so that only the following tags are processed.

  <p> <br> <em> <strong> <cite> <blockquote> <ul> <ol> <li>

Although not part of this security problem, we purposely disallowed <a> tags, deciding that they are not essential for comments on our site and almost always only appear in spam comments. As part of eliminating links, we also turned off Convert URLs Into Links.

Note that the user can still type in whatever they want, including any kind of HTML. The original text is stored unchanged as the content of the comment.

However, what the Filtered HTML format does is effectively strip out disallowed tags as part of the rendering process.

 - - - - -

Even with Filtered HTML, I was concerned that the onmouseover attribute could still be used inside one of the tags that is allowed. What about something like

  <p onmouseover="....">

But some testing and some stepping through core code revealed that onmouseover as well as other risky attributes are stripped out of those tags for output even though the tags themselves are processed.

The holes were plugged for these two exploits.

 - - - - -

The final thing left to do was to go back and check again the malicious comments, which I had left in the database while carrying out the above steps. I thought that applying Filtered HTML would neutralize them. I was wrong.

In the database table field_data_comment_body there is a column comment_body_format that stores the format in effect when the comment was created. The format apparently is used to render the comment even if it is no longer allowed.

As a result, those comments were still redirecting!

So I used SQL queries to find their entity ids, then deleted them by browsing to, for example, the following path where 30259 is the entity id of a comment.

  /comment/30259/edit

or

  /comment/30259/delete

The latter path goes directly to the delete confirmation dialog. In some cases I had to use it because loading the edit form for the comment triggered the redirection, so I couldn't even edit it.

 - - - - -

Michael Richardson at Pantheon support did an awesome job of doing the initial troubleshooting when I was at a loss as to where to start. It was he who uncovered the use of onmouseover on our site. Thanks, Michael!


Sources:

What is the Meta Refresh Tag?
http://webdesign.about.com/od/metataglibraries/a/aa080300a.htm

HTML onmouseover Event Attribute
http://www.w3schools.com/tags/ev_onmouseover.asp

Drupal Text Formats and Filters Tutorial
https://www.hostknox.com/tutorials/drupal/formats-and-filters

Text Filters and Input Formats
https://www.drupal.org/node/213156

Shareable URLs from Forward and the Twilio service: accessing an API endpoint on a localhost site

How do you have an external service access an API endpoint that is defined on a site that you're running locally?

One of the Drupal 7 sites (AllGoodText.com) I work on sends out SMS messages to users who register for weekly messages to be delivered to their cellphones.

The site uses the Twilio service to configure a virtual cellphone number from which the messages are apparently sent.

We also use the D7 Twilio module (version 7.x-1.10) to interface with the Twilio API. The module provides Drupal-specific elements such as hooks for implementing the desired functionality.

During local development, this worked fine as long as the site was only calling the Twilio API endpoints.

However, we wanted to enhance the site to define its own endpoints for the Twilio service to call back. For example, we wanted to be notified of and to be able to process STOP messages that our users were texting to our virtual cell number so that we could update the site's database.

For development purposes, I wanted to test as realistically as possible. I wanted my local site to interact with my cellphone via Twilio.

The problem was, I only run Apache on my system as a local server. At first, I thought I'd have to use our development site (it happens to be on Pantheon), constantly pushing code changes to it. That would have been tedious and messy.

Because I find the use of a debugger indispensable, I also would have had to figure out how to do so remotely.

Luckily, I saw a comment in response to a question about how to test the use of Twilio that mentioned "request forwarding utilities", of which Forward is one. The comment is clear and concise, so I'm just going to quote the commenter, Devin Rader.

Basically, request forwarding utilities like ForwardHQ do two things:

1. It creates a publicly accessible URL, that you can tell Twilio about.
2. It creates an SSH tunnel between a port on your machine and a server on the internet

When Twilio needs to make an HTTP request because it received an inbound SMS or voice call, it will request the ForwardHQ URL. Forward knows how to take that request and send it to the port running on your local machine. That port maps to your local web development server running in order to process the request and return the result to Twilio.

This sounded like exactly what I wanted. Here are the steps I took.

1.  Signed up for an account with Forward.

2.  Installed their browser extension for Chrome, which puts an icon on the toolbar to Open Forward.

3.  Browsed to the local site. Clicked on the Open Forward icon.

(I probably had to log in to Forward the first time I did this.)

4.  This brought up a special page from the browser extension. Clicked on Start Tunnel to create a tunnel between the site running on my localhost and Forward.

To access the site, the url wholewhale.fwd.wf was provided by default, where the subdomain wholewhale was specified by me.

5.  The Twilio module defines the path /twilio/sms using hook_menu() for processing incoming sms messages.

6.  So under my Twilio account, I specified http://wholewhale.fwd.wf/twilio/sms as the Request URL to notify the local site of incoming sms messages from users.

(On the live site, the endpoint will be http://allgoodtext.com/twilio/sms)

7.  I had an issue with the Twilio module logging the error message "Incoming SMS could not be validated" and not invoking the hook I had implemented.

I hacked around this by temporarily inserting a "return TRUE" statement in the module code so that it would continue as though no error had occurred. This worked fine for what I was doing.
 
I was now able to run the site and make code changes to it locally, run my debugger locally, and have the site interact with the Twilio service in both directions.

Sweet!!


Sources:

Forward
https://forwardhq.com/

twilio (the service)
https://www.twilio.com/

Twilio (the Drupal module)
https://www.drupal.org/project/twilio

How to test Twilio calling my REST API via POST when a user sends an sms to a short code?
http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh

Accessing localhost from Anywhere
http://www.sitepoint.com/accessing-localhost-from-anywhere/

Beta 11 --> Beta 12: _format property in route definition

Migrating to Beta 12, some ajax functionality was not functioning as expected. The log messages reported an exception,

Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException: No route found for the specified format

Using a debugger to step through the function that throws the exception, I made guesses as to what was needed. Eventually, I stumbled on a fix.

In the .routing.yml file I changed the relevant route definition by removing the _format property as follows.

Before:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely
    _format: json

After:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely

Unfortunately, searching through the Drupal and the Symfony docs, I have not found anything to help me understand why this change to the route definition makes a critical difference.

From the module's point of view, it looks as though the property is actually not needed. The client-side JavaScript that makes the request to the server sets json as the dataType. And the server-side code always instantiates a JsonResponse object to pass back.

Beta 11 --> Beta 12: Leading slash required in paths for "Add Alias" page (and elsewhere)

The Add Alias page that can be accessed via admin/config/search/path/add  now requires that values entered for both of the fields Existing System Path and Path Alias must start with a leading slash. Otherwise, the form does not validate.

When filling in the form manually, error messages are displayed to that effect, so it's clear what the problem is.

However, when this is being done programmatically in an automated Testing case, it's not at all obvious what went wrong.

Here's a piece of original code to create a path alias to an existing node, written for a subclass of WebTestBase:

  $edit = array();
  $edit['source'] = 'node/' . $node->id();
  $edit['alias'] = $this->randomMachineName(10);
  $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));

 And here's the corrected code:

  $edit = array();
  $edit['source'] = '/node/' . $node->id();
  $edit['alias'] = '/' . $this->randomMachineName(10);
  $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));

---

There seems to be an overall pattern that paths relative to the site root must now start with a slash. Besides these fields for the Add Alias page, I wrote in a previous post about such a requirement in calls to some methods of class AliasManager. And there's recent mention of this in Update the DB dump to have a leading slash for the frontpage.

Beta 11 --> Beta 12: AliasManager expects leading slash

As soon as I enabled the Optimizely module under Beta 12, I'd get the normal setup messages from the module, but then followed by this error.

The website encountered an unexpected error. Please try again later.

There was no other indication on the page of what had gone wrong. Moreover, the site would be completely unusable such that I could not even browse to the front page.

Eventually, I stumbled on a way to access the log. By deleting the directory for the module after the error occurred and using drush to clear cache, the site became usable again.

The log showed this message:

InvalidArgumentException: Source path node has to start with a slash. in Drupal\Core\Path\AliasManager->getAliasByPath() (line 191 of /var/www/html/opti/core/lib/Drupal/Core/Path/AliasManager.php).

This made me realize that the parameter in my calls to getAliasByPath() has to start with a leading slash. This also applies to the sibling method getPathByAlias().

So, for example, I revised the following function,

trait LookupPath  {

  static function lookupPathAlias($path) {

    $alias = \Drupal::service('path.alias_manager')->getAliasByPath($path); 
    return (strcmp($alias, $path) == 0) ? FALSE : $alias;
  }
} 


to call a new helper routine.

trait LookupPath  {

  static function lookupPathAlias($path) {

    $path = LookupPath::checkPath($path);
    $alias = \Drupal::service('path.alias_manager')->getAliasByPath($path);
    return (strcmp($alias, $path) == 0) ? FALSE : $alias;
  }

  static function checkPath($path) {
    return ($path[0] == '/') ? $path : '/' . $path;
  }
}


The fix was simple enough once I saw the error message in the log. The difficulty lay in getting to the log in the first place!