Shareable URLs from Forward and the Twilio service: accessing an API endpoint on a localhost site

How do you have an external service access an API endpoint that is defined on a site that you're running locally?

One of the Drupal 7 sites (AllGoodText.com) I work on sends out SMS messages to users who register for weekly messages to be delivered to their cellphones.

The site uses the Twilio service to configure a virtual cellphone number from which the messages are apparently sent.

We also use the D7 Twilio module (version 7.x-1.10) to interface with the Twilio API. The module provides Drupal-specific elements such as hooks for implementing the desired functionality.

During local development, this worked fine as long as the site was only calling the Twilio API endpoints.

However, we wanted to enhance the site to define its own endpoints for the Twilio service to call back. For example, we wanted to be notified of and to be able to process STOP messages that our users were texting to our virtual cell number so that we could update the site's database.

For development purposes, I wanted to test as realistically as possible. I wanted my local site to interact with my cellphone via Twilio.

The problem was, I only run Apache on my system as a local server. At first, I thought I'd have to use our development site (it happens to be on Pantheon), constantly pushing code changes to it. That would have been tedious and messy.

Because I find the use of a debugger indispensable, I also would have had to figure out how to do so remotely.

Luckily, I saw a comment in response to a question about how to test the use of Twilio that mentioned "request forwarding utilities", of which Forward is one. The comment is clear and concise, so I'm just going to quote the commenter, Devin Rader.

Basically, request forwarding utilities like ForwardHQ do two things:

1. It creates a publicly accessible URL, that you can tell Twilio about.
2. It creates an SSH tunnel between a port on your machine and a server on the internet

When Twilio needs to make an HTTP request because it received an inbound SMS or voice call, it will request the ForwardHQ URL. Forward knows how to take that request and send it to the port running on your local machine. That port maps to your local web development server running in order to process the request and return the result to Twilio.

This sounded like exactly what I wanted. Here are the steps I took.

1.  Signed up for an account with Forward.

2.  Installed their browser extension for Chrome, which puts an icon on the toolbar to Open Forward.

3.  Browsed to the local site. Clicked on the Open Forward icon.

(I probably had to log in to Forward the first time I did this.)

4.  This brought up a special page from the browser extension. Clicked on Start Tunnel to create a tunnel between the site running on my localhost and Forward.

To access the site, the url wholewhale.fwd.wf was provided by default, where the subdomain wholewhale was specified by me.

5.  The Twilio module defines the path /twilio/sms using hook_menu() for processing incoming sms messages.

6.  So under my Twilio account, I specified http://wholewhale.fwd.wf/twilio/sms as the Request URL to notify the local site of incoming sms messages from users.

(On the live site, the endpoint will be http://allgoodtext.com/twilio/sms)

7.  I had an issue with the Twilio module logging the error message "Incoming SMS could not be validated" and not invoking the hook I had implemented.

I hacked around this by temporarily inserting a "return TRUE" statement in the module code so that it would continue as though no error had occurred. This worked fine for what I was doing.
 
I was now able to run the site and make code changes to it locally, run my debugger locally, and have the site interact with the Twilio service in both directions.

Sweet!!


Sources:

Forward
https://forwardhq.com/

twilio (the service)
https://www.twilio.com/

Twilio (the Drupal module)
https://www.drupal.org/project/twilio

How to test Twilio calling my REST API via POST when a user sends an sms to a short code?
http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh

Accessing localhost from Anywhere
http://www.sitepoint.com/accessing-localhost-from-anywhere/

Beta 11 --> Beta 12: _format property in route definition

Migrating to Beta 12, some ajax functionality was not functioning as expected. The log messages reported an exception,

Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException: No route found for the specified format

Using a debugger to step through the function that throws the exception, I made guesses as to what was needed. Eventually, I stumbled on a fix.

In the .routing.yml file I changed the relevant route definition by removing the _format property as follows.

Before:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely
    _format: json

After:

ajax.enable:
  path: /ajax/optimizely
  defaults:
    _controller: \Drupal\optimizely\AjaxEnable::enableDisable
    _title: Optimizely Administer AJAX
  requirements:
    _permission: administer optimizely

Unfortunately, searching through the Drupal and the Symfony docs, I have not found anything to help me understand why this change to the route definition makes a critical difference.

From the module's point of view, it looks as though the property is actually not needed. The client-side JavaScript that makes the request to the server sets json as the dataType. And the server-side code always instantiates a JsonResponse object to pass back.

Beta 11 --> Beta 12: Leading slash required in paths for "Add Alias" page (and elsewhere)

The Add Alias page that can be accessed via admin/config/search/path/add  now requires that values entered for both of the fields Existing System Path and Path Alias must start with a leading slash. Otherwise, the form does not validate.

When filling in the form manually, error messages are displayed to that effect, so it's clear what the problem is.

However, when this is being done programmatically in an automated Testing case, it's not at all obvious what went wrong.

Here's a piece of original code to create a path alias to an existing node, written for a subclass of WebTestBase:

  $edit = array();
  $edit['source'] = 'node/' . $node->id();
  $edit['alias'] = $this->randomMachineName(10);
  $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));

 And here's the corrected code:

  $edit = array();
  $edit['source'] = '/node/' . $node->id();
  $edit['alias'] = '/' . $this->randomMachineName(10);
  $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));

---

There seems to be an overall pattern that paths relative to the site root must now start with a slash. Besides these fields for the Add Alias page, I wrote in a previous post about such a requirement in calls to some methods of class AliasManager. And there's recent mention of this in Update the DB dump to have a leading slash for the frontpage.

Beta 11 --> Beta 12: AliasManager expects leading slash

As soon as I enabled the Optimizely module under Beta 12, I'd get the normal setup messages from the module, but then followed by this error.

The website encountered an unexpected error. Please try again later.

There was no other indication on the page of what had gone wrong. Moreover, the site would be completely unusable such that I could not even browse to the front page.

Eventually, I stumbled on a way to access the log. By deleting the directory for the module after the error occurred and using drush to clear cache, the site became usable again.

The log showed this message:

InvalidArgumentException: Source path node has to start with a slash. in Drupal\Core\Path\AliasManager->getAliasByPath() (line 191 of /var/www/html/opti/core/lib/Drupal/Core/Path/AliasManager.php).

This made me realize that the parameter in my calls to getAliasByPath() has to start with a leading slash. This also applies to the sibling method getPathByAlias().

So, for example, I revised the following function,

trait LookupPath  {

  static function lookupPathAlias($path) {

    $alias = \Drupal::service('path.alias_manager')->getAliasByPath($path); 
    return (strcmp($alias, $path) == 0) ? FALSE : $alias;
  }
} 


to call a new helper routine.

trait LookupPath  {

  static function lookupPathAlias($path) {

    $path = LookupPath::checkPath($path);
    $alias = \Drupal::service('path.alias_manager')->getAliasByPath($path);
    return (strcmp($alias, $path) == 0) ? FALSE : $alias;
  }

  static function checkPath($path) {
    return ($path[0] == '/') ? $path : '/' . $path;
  }
}


The fix was simple enough once I saw the error message in the log. The difficulty lay in getting to the log in the first place!

"The RSA host key for git.drupal.org has changed"

Just now when I tried to git push to drupal.org I got a long warning message that started with the following.

The RSA host key for git.drupal.org has changed, and the key for the corresponding IP address 140.211.10.43 is unknown.

The article  Drupal.org Git Server Migration  and its comments give a good explanation and discussion about how drupal.org has migrated to different servers as of early July.

The steps I took to fix the issue on my Linux Mint system were as follows, where the -R option for ssh-keygen means "remove the keys".

# ssh-keygen -R git.drupal.org
# ssh-keygen -R 140.211.10.43

# git fetch

The authenticity of host 'git.drupal.org (140.211.10.43)' can't be established.

RSA key fingerprint is 16:f5:44:6c:a1:c6:be:72:cd:98:b5:b7:7d:26:d6:14.

Are you sure you want to continue connecting (yes/no)? yes


After that, there were no further warnings in using git to access drupal.org.

Instead of git fetch probably any git command that accesses the remote repo would have the same effect, such as git pull or git clone.

Source:

Drupal.org Git Server Migration
https://www.drupal.org/node/2529356