Dec 12, 2015

Drupal 8, Beta 14 --> Beta 15: Missing langcode in configuration schema

In migrating from Drupal 8 beta 14 to beta 15, the functionality of the module itself worked fine, but some of the automated tests were failing with error messages that included the following.
Uncaught PHP Exception Drupal\Core\Config\Schema\SchemaIncompleteException: "Schema errors for optimizely.settings with the following errors: optimizely.settings:langcode missing schema" at /var/www/html/opti/core/lib/Drupal/Core/Config/Testing/ConfigSchemaChecker.php line 98
In this message, optimizely.settings is the name of a group of configuration settings. It is also a configuration key in a .schema.yml file that is required for automated testing, which I blogged about earlier at Beta 3 --> Beta 4:  Configuration schema and metadata.

The article Fix config schema mentioned a very similar error message and stated "Because of a recent core change all tests are failing". I looked at the patches in that article, but I could not figure out what needed to be added in our case.

Then a search through the core code for the string "langcode:" led me to add langcode: as a key to the .schema.yml file as in the following.

optimizely.settings:
  type: mapping
  label: 'Optimizely Config Data'
  mapping:
    optimizely_id:
      type: integer
      label: 'Optimizely ID Number'
      translatable: false
    langcode:
      type: string
      label: 'Language code'


Problem solved. All automated tests passed after this change.

Update:  Also see my later post on  Module: Configuration Inspector for Drupal 8.

Sources:

Beta 3 --> Beta 4: Configuration schema and metadata
http://optimizely-to-drupal-8.blogspot.com/2014/12/beta-3-beta-4-configuration-schema-and.html

Fix config schema
https://www.drupal.org/node/2547365

All TestBase derived tests now enforce strict configuration schema adherence by default
https://www.drupal.org/node/2391795

Configuration schema/metadata
https://www.drupal.org/node/1905070

Nov 10, 2015

"Notice: Undefined index: und in eval() (line . . . . . /modules/php/php.module(80) : eval()'d code)"

On my local instance of a Drupal 7 site, I'd get warnings like the following on almost every page for a custom content type called Poem.

Notice: Undefined index: und in eval() (line 12 of /var/www/html/power-poetry/modules/php/php.module(80) : eval()'d code).

Although most probably innocuous, these warnings were also being logged numerous times in the system log, cluttering it up and making it harder to spot other, significant messages. It was enough of an annoyance that I decided to fix it.

I was able  to track this to a snippet of PHP code that is used in a Drupal block of ours. The code is part of the block's Visibility Settings. It determines if the current Poem being rendered satisfies certain criteria or not. If so, the block is made visible.

The offending line turned out to be

  $slam_id = $current_slam['und'][0]['target_id'];

The variable $current_slam was often an empty array. So by replacing the line with an additional check, the PHP warnings disappeared.

  if (!empty($current_slam)) {
    $slam_id = $current_slam['und'][0]['target_id'];
  }

The key to finding this quickly was remembering that we have this user-supplied snippet of code that needs to be evaluated at runtime

  - - - - -

While debugging, I wanted to see the type and the value of the variable $current_slam. The following worked to output it to the system log.

  watchdog('debug', print_r($current_slam, TRUE), array(),
           WATCHDOG_NOTICE);

The second parameter to print_r() determines what kind of return value the function passes back. By default, print_r() returns its status. But by setting the second param to TRUE, the return value is the output string instead.

  - - - - -

The site is hosted on Pantheon, which provides this status message:

  • PHP Filter: PHP Filter is enabled! Executable code should never be stored in the database, and support for this feature was removed in Drupal 8 - https://drupal.org/node/1203886
    Remove all executable code from your content and move it to your codebase.

Source:

Remove the PHP module from Drupal core
https://drupal.org/node/1203886

Nov 3, 2015

Malicious page redirects and Drupal's Filtered-HTML text format

One of the Drupal 7 sites I maintain is www.powerpoetry.org which is a platform for publishing poems and comments about them.

Recently, we heard from an irate poet. When browsing to one of her poems, the page would partially load, then automatically redirect to a movie streaming site that had nothing to do with her poem nor with the site in general. Her poem page somehow got hijacked, and readers could never read her work.

It turned out that all site users, even Anonymous ones, had permissions for the Full HTML Text Format. This allowed comments to be submitted that contained malicious code to redirect the page.

So far, I have found two different techniques that were used to implement redirection.

(1)  Use of the <meta> tag with an attribute of http-equiv="refresh". For example,

  <meta http-equiv="refresh" 
        content="2;url=http://scumbags.com/">

where the value of 2 means a delay of two seconds before the refresh.

(2)  Use of the onmouseover attribute to execute JavaScript. For example,

<a href="//tinyurl.com/nm8ugh
   onmouseover="document.location='//tinyurl.com/nm8ugh';">

As documented, this attribute and closely related ones are valid for almost all HTML tags, which would make it potentially a big problem where markup is allowed.

If you have good knowledge of HTML, these techniques may be obvious, but for me as a back-end developer they were new.

Because I have direct access to the Drupal database, I was able to use a SQL query such as the following to find malicious comments.

  select entity_id, comment_body_value
    from field_data_comment_body
    where comment_body_value like "%onmouseover%";

 - - - - -

To help plug these security holes, we decided to allow only Administrators to have access to all Text Formats. Depending on the needs of your site, you might want to disable the Full HTML format entirely.

For all other users, the Filtered HTML Text Format was permitted, configured using Limit Allowed HTML Tags so that only the following tags are processed.

  <p> <br> <em> <strong> <cite> <blockquote> <ul> <ol> <li>

Although not part of this security problem, we purposely disallowed <a> tags, deciding that they are not essential for comments on our site and almost always only appear in spam comments. As part of eliminating links, we also turned off Convert URLs Into Links.

Note that the user can still type in whatever they want, including any kind of HTML. The original text is stored unchanged as the content of the comment.

However, what the Filtered HTML format does is effectively strip out disallowed tags as part of the rendering process.

 - - - - -

Even with Filtered HTML, I was concerned that the onmouseover attribute could still be used inside one of the tags that is allowed. What about something like

  <p onmouseover="....">

But some testing and some stepping through core code revealed that onmouseover as well as other risky attributes are stripped out of those tags for output even though the tags themselves are processed.

The holes were plugged for these two exploits.

 - - - - -

The final thing left to do was to go back and check again the malicious comments, which I had left in the database while carrying out the above steps. I thought that applying Filtered HTML would neutralize them. I was wrong.

In the database table field_data_comment_body there is a column comment_body_format that stores the format in effect when the comment was created. The format apparently is used to render the comment even if it is no longer allowed.

As a result, those comments were still redirecting!

So I used SQL queries to find their entity ids, then deleted them by browsing to, for example, the following path where 30259 is the entity id of a comment.

  /comment/30259/edit

or

  /comment/30259/delete

The latter path goes directly to the delete confirmation dialog. In some cases I had to use it because loading the edit form for the comment triggered the redirection, so I couldn't even edit it.

 - - - - -

Michael Richardson at Pantheon support did an awesome job of doing the initial troubleshooting when I was at a loss as to where to start. It was he who uncovered the use of onmouseover on our site. Thanks, Michael!


Sources:

What is the Meta Refresh Tag?
http://webdesign.about.com/od/metataglibraries/a/aa080300a.htm

HTML onmouseover Event Attribute
http://www.w3schools.com/tags/ev_onmouseover.asp

Drupal Text Formats and Filters Tutorial
https://www.hostknox.com/tutorials/drupal/formats-and-filters

Text Filters and Input Formats
https://www.drupal.org/node/213156

Sep 21, 2015

Shareable URLs from Forward and the Twilio service: accessing an API endpoint on a localhost site

How do you have an external service access an API endpoint that is defined on a site that you're running locally?

One of the Drupal 7 sites (AllGoodText.com) I work on sends out SMS messages to users who register for weekly messages to be delivered to their cellphones.

The site uses the Twilio service to configure a virtual cellphone number from which the messages are apparently sent.

We also use the D7 Twilio module (version 7.x-1.10) to interface with the Twilio API. The module provides Drupal-specific elements such as hooks for implementing the desired functionality.

During local development, this worked fine as long as the site was only calling the Twilio API endpoints.

However, we wanted to enhance the site to define its own endpoints for the Twilio service to call back. For example, we wanted to be notified of and to be able to process STOP messages that our users were texting to our virtual cell number so that we could update the site's database.

For development purposes, I wanted to test as realistically as possible. I wanted my local site to interact with my cellphone via Twilio.

The problem was, I only run Apache on my system as a local server. At first, I thought I'd have to use our development site (it happens to be on Pantheon), constantly pushing code changes to it. That would have been tedious and messy.

Because I find the use of a debugger indispensable, I also would have had to figure out how to do so remotely.

Luckily, I saw a comment in response to a question about how to test the use of Twilio that mentioned "request forwarding utilities", of which Forward is one. The comment is clear and concise, so I'm just going to quote the commenter, Devin Rader.
Basically, request forwarding utilities like ForwardHQ do two things:
  1. It creates a publicly accessible URL, that you can tell Twilio about.
  2. It creates an SSH tunnel between a port on your machine and a server on the internet
When Twilio needs to make an HTTP request because it received an inbound SMS or voice call, it will request the ForwardHQ URL. Forward knows how to take that request and send it to the port running on your local machine. That port maps to your local web development server running in order to process the request and return the result to Twilio.

This sounded like exactly what I wanted. Here are the steps I took.

1.  Signed up for an account with Forward.

2.  Installed their browser extension for Chrome, which puts an icon on the toolbar to Open Forward.

3.  Browsed to the local site. Clicked on the Open Forward icon.

(I probably had to log in to Forward the first time I did this.)

4.  This brought up a special page from the browser extension. Clicked on Start Tunnel to create a tunnel between the site running on my localhost and Forward.

To access the site, the url wholewhale.fwd.wf was provided by default, where the subdomain wholewhale was specified by me.

5.  The Twilio module defines the path /twilio/sms using hook_menu() for processing incoming sms messages.

6.  So under my Twilio account, I specified http://wholewhale.fwd.wf/twilio/sms as the Request URL to notify the local site of incoming sms messages from users.

(On the live site, the endpoint will be http://allgoodtext.com/twilio/sms)

7.  I had an issue with the Twilio module logging the error message "Incoming SMS could not be validated" and not invoking the hook I had implemented.

I hacked around this by temporarily inserting a "return TRUE" statement in the module code so that it would continue as though no error had occurred. This worked fine for what I was doing.
 
I was now able to run the site and make code changes to it locally, run my debugger locally, and have the site interact with the Twilio service in both directions.

Sweet!!


Sources:

Forward
https://forwardhq.com/

twilio (the service)
https://www.twilio.com/

Twilio (the Drupal module)
https://www.drupal.org/project/twilio

How to test Twilio calling my REST API via POST when a user sends an sms to a short code?
http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh

Accessing localhost from Anywhere
http://www.sitepoint.com/accessing-localhost-from-anywhere/

Aug 24, 2015

Beta 11 --> Beta 12:    _format property in route definition

Migrating to Beta 12, some ajax functionality was not functioning as expected. The log messages reported an exception,

Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException: No route found for the specified format

Using a debugger to step through the function that throws the exception, I made guesses as to what was needed. Eventually, I stumbled on a fix.

In the .routing.yml file I changed the relevant route definition by removing the _format property as follows.

Before:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely
    _format: json


After:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely


Unfortunately, searching through the Drupal and the Symfony docs, I have not found anything to help me understand why this change to the route definition makes a critical difference.

From the module's point of view, it looks as though the property is actually not needed. The client-side JavaScript that makes the request to the server sets json as the dataType. And the server-side code always instantiates a JsonResponse object to pass back.