tag:blogger.com,1999:blog-50234485428234477112023-12-29T11:33:37.490-08:00Optimizely Module: Notes on Converting to Drupal 8Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.comBlogger102125tag:blogger.com,1999:blog-5023448542823447711.post-23350101365559252542017-10-15T21:25:00.000-07:002017-10-15T21:25:54.792-07:00An Embarrassing Mistake: 'The "example" entity type does not exist'Using the code from <a href="https://www.drupal.org/docs/8/api/configuration-api/creating-a-configuration-entity-type-in-drupal-8"><i>Creating a configuration entity type in Drupal 8</i></a> to create a sample module for demonstrating use of configuration entities, I got the following log message when browsing to the path<span style="font-family: "courier new" , "courier" , monospace;"> /admin/config/system/example</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Drupal\Component\Plugin\Exception\PluginNotFoundException: The "example" entity type does not exist. in Drupal\Core\Entity\EntityTypeManager->getDefinition() (line 133 of /var/www/html/opti/core/lib/Drupal/Core/Entity/EntityTypeManager.php).</span><br />
<br />
This is embarrassing, but I'm going to write it up anyway.<br />
<br />
I had copied and pasted the code verbatim from the article into new files using the filenames and paths that were indicated. It should have worked!<br />
<br />
It turns out I had forgotten one little thing: at the top of each of the php files, I had neglected to insert<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><?php</span><br />
<br />
What eventually tipped me off was that my text editor, Sublime Text, was not providing the syntax coloring that it normally did.<br />
<br />
Laughing my ass off ...<br />
<br />
<br />
Sources:<br />
<br />
<i>Creating a configuration entity type in Drupal 8</i> <br />
<a href="https://www.drupal.org/docs/8/api/configuration-api/creating-a-configuration-entity-type-in-drupal-8">https://www.drupal.org/docs/8/api/configuration-api/creating-a-configuration-entity-type-in-drupal-8</a><br />
<br />
<i>Configuration Entities in Drupal 8</i><br />
<a href="https://wunder.io/blog/configuration-entities-in-drupal-8/2014-07-14">https://wunder.io/blog/configuration-entities-in-drupal-8/2014-07-14</a>Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-44481478943521546202017-09-02T10:38:00.001-07:002017-09-02T10:38:37.166-07:00How to enable Vagrant SSH access into a Docker containerVagrant seems to work really well with VirtualBox and some other virtualization providers, but I had a situation where I wanted to get it to run with Docker instead (on my local Linux Mint system) and to enable Vagrant to have ssh access into the Docker container.<br />
<br />
I started with the Docker image<span style="font-family: "courier new" , "courier" , monospace;"> wadmiraal/drupal:7.54 </span>which I had been using for doing Drupal development via Docker alone. There are several changes that had to be made, some in the Docker image, some in the Vagrantfile, in order to satisfy Vagrant expectations.<br />
<br />
<b>(1) Create a user named<span style="font-family: "courier new" , "courier" , monospace;"> vagrant </span>in the guest system</b>.<br />
<br />
Running the image in a Docker container, I<span style="font-family: "courier new" , "courier" , monospace;"> ssh</span>'ed into it using its existing root user, then created a new user "<span style="font-family: "courier new" , "courier" , monospace;">vagrant</span>". Set the password to "<span style="font-family: "courier new" , "courier" , monospace;">vagrant</span>", which is a convention but it could be a different password.<br />
<br />
I tested this new user by logging in manually.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># ssh -p 2222 vagrant@127.0.0.1</span><br />
<br />
where 2222 is the local host port that is forwarded to the ssh port on the guest.<br />
<br />
<b>(2) Provide ssh key authentication in the guest system.</b><br />
<br />
On the host / local system, I already had ssh keys. I used the following command to set up the key in the guest system.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># ssh-copy-id -p 2222 vagrant@127.0.0.1</span><br />
<br />
Again, I tested this by logging in manually.<br />
<br />
<b>(3) Enable<span style="font-family: "courier new" , "courier" , monospace;"> sudo </span>for the vagrant user</b><b><b> in the guest system</b>.</b><br />
<br />
The Vagrant docs say, "Many aspects of Vagrant expect the default SSH user to have passwordless sudo configured. This lets Vagrant configure networks, mount synced folders, install software, and more." <br />
<br />
The Docker image I started with did not even have<span style="font-family: "courier new" , "courier" , monospace;"> sudo </span>available, so I logged into the guest as<span style="font-family: "courier new" , "courier" , monospace;"> root </span>and installed it.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># apt-get update</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># apt-get install sudo</span><br />
<br />
Then used<span style="font-family: "courier new" , "courier" , monospace;"> visudo </span>to edit<span style="font-family: "courier new" , "courier" , monospace;"> /etc/sudoers </span>to insert two lines.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># visudo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">vagrant ALL=(ALL) NOPASSWD:ALL<br />Defaults:vagrant !requiretty</span><br />
<br />
After making these changes to the guest, I used Docker to commit a new image. Let's call it<span style="font-family: "courier new" , "courier" , monospace;"> earl/drupal:7.54</span><br />
<br />
<b>(4) Add Docker settings</b><b><b> in the Vagrantfile</b>.</b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">ENV['VAGRANT_DEFAULT_PROVIDER'] = 'docker'</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Vagrant.configure("2") do |config|</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /> config.vm.network "forwarded_port", guest: 22, host: 2222<br /><br /> config.vm.provider "docker" do |dock|<br /> dock.image = "earl/drupal:7.54"<br /> dock.name = "drupal-7.54"<br /> dock.has_ssh = true<br /> end</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">end</span><br />
<br />
Specifying the default provider in the Vagrantfile is just a convenience so that you don't have to use the<span style="font-family: "courier new" , "courier" , monospace;"> --provider </span>option for the<span style="font-family: "courier new" , "courier" , monospace;"> vagrant up </span>command.<br />
<br />
<b>(5) Set up password authentication</b><b><b> for ssh</b></b><b><b><b> in the Vagrantfile</b></b>.</b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Vagrant.configure("2") do |config|</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> config.ssh.username = "vagrant"<br /> config.ssh.password = "vagrant"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">end</span><br />
<br />
This is optional. If you provide<span style="font-family: "courier new" , "courier" , monospace;"> config.ssh.password </span>as above, Vagrant will use password authentication. Otherwise, Vagrant will default to key authentication, as in the following step.<b> </b><br />
<br />
<b>(6) Set up key authentication</b><b><b> for ssh</b></b><b><b><b><b><b> in the Vagrantfile</b></b></b></b>.</b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Vagrant.configure("2") do |config|</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> config.ssh.keys_only = false<br /> config.ssh.private_key_path = "/home/earl/.ssh/id_rsa"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">end</span><br />
<br />
This is also optional. Do either step (5) or step (6). If you do both, Vagrant will use password authentication.<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">ssh.keys_only </span>must be set to<span style="font-family: "courier new" , "courier" , monospace;"> false </span>in order to use your own ssh keys, and you must also provide the path to those keys.<br />
<br />
<b>(7) Test that Vagrant is able to make a change in the guest</b><b>.</b><br />
<br />
For example, I added the following two lines to the Vagrantfile to execute a shell command.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Vagrant.configure("2") do |config|</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> config.vm.provision "shell",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> inline: "touch /vagrant/hello-world"</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">end</span><br />
<br />
Doing<span style="font-family: "courier new" , "courier" , monospace;"> vagrant up </span>should boot up without errors and execute the shell command successfully. In the above snippet, because Vagrant automatically synchronizes the<span style="font-family: "courier new" , "courier" , monospace;"> /vagrant </span>directory on the guest with the host directory where the Vagrantfile is located, you can check for the<span style="font-family: "courier new" , "courier" , monospace;"> hello-world </span>file on the host.<br />
<br />
Sources:<br />
<br />
<i>Creating a Base Box</i><br />
<a href="https://www.vagrantup.com/docs/boxes/base.html">https://www.vagrantup.com/docs/boxes/base.html</a><br />
<br />
<i>SSH Settings</i><br />
<a href="https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html">https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html</a><br />
<br />
<i>Docker Configuration</i><br />
<a href="https://www.vagrantup.com/docs/docker/configuration.html">https://www.vagrantup.com/docs/docker/configuration.html</a> Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-92212328349678733762017-08-25T09:58:00.000-07:002017-08-25T10:00:20.094-07:00Moving Docker's storage to a different locationOn a Linux Mint system, I was running low on disk space in the partition where Docker CE 17.06 had installed itself and was storing its files such as images. So I thought I'd move the<span style="font-family: "courier new" , "courier" , monospace;"> docker </span>storage directory to a different partition that had a lot more space, and create a symbolic link in the directory's original location to where it had been moved.<br />
<br />
That sounded pretty simple, but it turned out to be a vexing problem.<br />
<br />
After moving the directory, I ran a Docker container that contained an instance of Drupal 7. When I browsed to the Drupal site I got the error:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'</span></blockquote>
If you do a web search on this error <span style="font-family: inherit;">message</span>, there are a lot of possible causes. In my case, the Docker container had been running fine just before the directory move, so that was an obvious culprit.<br />
<br />
I had typed in several<span style="font-family: "courier new" , "courier" , monospace;"> cp</span><span style="font-family: "courier new" , "courier" , monospace;"> <span style="font-family: inherit;"></span></span><span style="font-family: inherit;"></span>commands before running the container again, but I didn't have a clear memory or record of the steps I had taken. After the first occurrence of the error, I did further copy commands. The error kept occurring, but once in a while the site came up okay. This thrashing about included my re-initializing the<span style="font-family: "courier new" , "courier" , monospace;"> docker </span>directory at least once.<br />
<br />
From my frustrating experience, here are a few suggestions about moving Docker's storage.<br />
<br />
(a) Be sure to stop the Docker service before making any changes.<br />
<br />
(b) Use Docker's configuration file and the<span style="font-family: "courier new" , "courier" , monospace;"> -g </span>option to indicate the location of storage. This provides both flexibility and safety in trying different locations.<br />
<br />
(c) If you use the<span style="font-family: "courier new" , "courier" , monospace;"> cp </span>command to copy storage contents, be sure to include the<span style="font-family: "courier new" , "courier" , monospace;"> -p </span>option to preserve owner, mode, and timestamp.<br />
<br />
<hr width="90%" />
<br />
Here's an example of a sequence of steps that has worked in moving the Docker storage folder to a different location by using<span style="font-family: "courier new" , "courier" , monospace;"> cp</span>.<br />
<br />
(1) Stop the Docker service.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># service docker stop</span><br />
<br />
(2) Copy Docker's storage folder to a different partition.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># cd /var/lib</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># cp -r -p docker /home/earl</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># mv docker docker-save </span><br />
<br />
(3) Edit the Docker configuration file<span style="font-family: "courier new" , "courier" , monospace;"> </span>to add the<span style="font-family: "courier new" , "courier" , monospace;"> -g </span>option to point to the new location. The file may already contain a commented line you can use as a starting point.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># vim</span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> /etc/default</span>/docker</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span><span style="font-family: "courier new" , "courier" , monospace;">DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /home/earl/docker" </span><br />
<br />
(4) Re-start the Docker service. Run your container to test the change.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># service docker start </span><br />
<br />
(5) When you are confident that the change is correct, you can remove the original directory to recover the space.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># rm -r </span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">/var/lib/</span>docker-save</span><br />
<br />
Sources:<br />
<br />
<i>How do I change the Docker image installation directory?</i><br />
<a href="https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169">https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169</a>Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-47979970712425496472017-08-02T16:24:00.000-07:002017-08-02T16:24:10.401-07:00VirtualBox, Vagrant, and KVMI was exploring the possibility of using Vagrant with VirtualBox for doing development on my local system.<br />
<br />
The system is running Linux Mint 17, and I was using Vagrant 1.9.7 with VirtualBox 5.1.24.<br />
<br />
I immediately starting having problems with some of the Vagrant boxes that I tried to run. (A "Vagrant box" is an initial machine image to be loaded into the virtual machine.)<br />
<br />
For example, using the box<span style="font-family: "courier new" , "courier" , monospace;"> hashicorp/precise64</span>, the<span style="font-family: "courier new" , "courier" , monospace;"> vagrant up </span>command showed the error:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Stderr: VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX) <br />VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole </span></blockquote>
Later, after I had learned how to configure Vagrant to have VirtualBox display its own user window, I tried to run the box<span style="font-family: "courier new" , "courier" , monospace;"> geerlingguy/ubuntu1604 </span>and got this error from VirtualBox:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">VT-x/AMD-V hardware acceleration is not available on your system. Your 64-bit guest will fail to detect a 64-bit CPU and will not be able to boot.</span> </blockquote>
And on the command line,<span style="font-family: "Courier New",Courier,monospace;"> kvm-ok </span>showed:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">INFO: Your CPU does not support KVM extensions </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">KVM acceleration can NOT be used </span><span style="font-family: "courier new" , "courier" , monospace;"></span></blockquote>
It turns out that there are two possible explanations for these error messages.<br />
<br />
(1) The processor hardware does not have the capabilities required by VirtualBox to support Linux KVM (kernel-based virtual machines). These capabilities are either Intel's <i>VT-x</i> or AMD's <i>AMD-V.</i><br />
<br />
VT-x is sometimes encoded as<span style="font-family: "courier new" , "courier" , monospace;"> vmx</span>, and AMD-V is sometimes encoded as<span style="font-family: "courier new" , "courier" , monospace;"> svm</span>.<br />
<br />
(2) The processor hardware has the capabilities but they are not enabled.<br />
<br />
<i>Without this hardware, VirtualBox cannot run 64-bit operating systems. However, it can still run 32-bit operating systems.</i><br />
<br />
In my case, it turned out that I have a lower-end processor that does not have the capabilities at all. I was able to check this by looking at the file<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: "Courier New",Courier,monospace;"> /proc/cpuinfo</span> </span>to find the model number of the processor:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> model name : Intel(R) Pentium(R) CPU B960 @ 2.20GHz</span><br />
<br />
Then using the model number<span style="font-family: "courier new" , "courier" , monospace;"> B960 </span>I searched <a href="https://ark.intel.com/#@Processors">this Intel site</a> to find its specs.<br />
<br />
If the processor has the capabilities but they are not enabled, then you might be able to enable them by going into the BIOS and looking for a setting such as VT (virtualization technology).<br />
<br />
And for my <i>next</i> system, I will be looking for the processor to have this.<br />
<br />
Sources:<br />
<br /><i>ERROR: VT-X is not available</i><br /><a href="https://forums.virtualbox.org/viewtopic.php?f=8&t=17090">https://forums.virtualbox.org/viewtopic.php?f=8&t=17090</a><br /><br /><i>PRODUCT SPECIFICATIONS</i><br /><a href="https://ark.intel.com/#@Processors">https://ark.intel.com/#@Processors</a><br /><br /><i>KVM/Installation</i><br />
<a href="https://help.ubuntu.com/community/KVM/Installation">https://help.ubuntu.com/community/KVM/Installation</a><br /><br /><i>x86 virtualization</i><br /><a href="https://en.wikipedia.org/wiki/X86_virtualization">https://en.wikipedia.org/wiki/X86_virtualization</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-58365536834041270342017-05-25T17:31:00.000-07:002017-05-25T17:31:06.498-07:00A Reason to Use the Drupal Coder / PHP_CodeSniffer utility<a href="https://www.drupal.org/project/coder">Drupal Coder</a> includes the command line utility <a href="https://pear.php.net/manual/en/package.php.php-codesniffer.php">PHP_CodeSniffer</a>, which parses source code to detect violations of a coding standard.<br />
<br />
Actually, there are two utilities in the package, one to detect violations, a second to automatically make changes for those that can be so fixed. <br />
<br />
I used the provided<span style="font-family: "courier new" , "courier" , monospace;"> Drupal </span>standard and ran it against all of the source code of the Optimizely module.<br />
<br />
There are more than a hundred "sniffs" that are checked against. Individually, many are minor and relatively insignificant. A few are downright annoying. In the aggregate, though, I do feel that the resulting code was improved in terms of its readability.<br />
<br />
The main benefit I've experienced so far is that I am being nudged into writing doc comments for all functions and classes. At first, I was resistent to doing so because good naming is often sufficient as documentation. As I edited file after file, though, I started to appreciate this kind of commenting as a desirable, consistent practice to adopt.<br />
<br />
In one case, writing descriptions about a class and its methods helped me realize that the class was not entirely cohesive and maybe should have been written as two classes instead.<br />
<br />
Retroactively applying the Drupal coding standard to the entire module was quite a bit of work. Moving forward, using PHP_CodeSniffer incrementally as a matter of habit should be much, much easier.<br />
<br />
Sources:<br />
<br />
<i>Coder</i> <br />
<a href="https://www.drupal.org/project/coder">https://www.drupal.org/project/coder</a><br />
<br />
<i>Installing Coder Sniffer</i><br />
<a href="https://www.drupal.org/node/1419988">https://www.drupal.org/node/1419988</a> Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-40954251945228773822017-04-21T11:29:00.000-07:002017-04-21T11:29:00.076-07:00Using xDebug and Sublime Text with DockerI've used xDebug with Sublime Text locally for quite some time but have started playing with Docker containers to instantiate instances of Apache, PHP, MySQL, and Drupal 8.<br />
<br />
The Docker
image I use has xDebug enabled for PHP, but I wanted to have xDebug running in the container to communicate with Sublime running on my local host system. This is not complicated, but it still took quite awhile for me to determine the correct settings.<br />
<br />
Some
of my confusion was due to the terms <i>server</i> and <i>client</i> as used in
documentation and comments. Most of the time, <i>server</i> refers to xDebug
running within PHP, and <i>client</i> refers to the IDE or text editor such as
Sublime.<br />
<br />
On the other hand, apparently it is xDebug that
initiates the connection to the IDE, which makes xDebug act like a
client. Also, xDebug has a setting called<span style="font-family: "courier new" , "courier" , monospace;"> remote_host </span>which sounds like a remote server that it is communicating with.<br />
<br />
In the container I'm running, the xDebug settings are in<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> /etc/php5/mods-available/xdebug.ini</span><br />
<br />
Here are tips to get these components working together.<br />
<br />
<span style="font-size: large;">• </span>Run<span style="font-family: "courier new" , "courier" , monospace;"> ifconfig </span>in a local terminal to get the local ip address.<br />
<br />
Working on my laptop connected to a home router,<span style="font-family: "courier new" , "courier" , monospace;"> ifconfig </span>shows<span style="font-family: "courier new" , "courier" , monospace;"> <span style="font-family: "courier new" , "courier" , monospace;">eth</span>0 </span>with<span style="font-family: "courier new" , "courier" , monospace;"> inet addr</span><span style="font-family: "courier new" , "courier" , monospace;"> 10.0.0.<span style="font-family: "courier new" , "courier" , monospace;">3</span></span>. I use that value as follows in <span style="font-family: "courier new" , "courier" , monospace;">xdebug.ini<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "courier new" , "courier" , monospace;"> </span></span></span></span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "courier new" , "courier" , monospace;"> xdebug.remote_host = 10.0.0.<span style="font-family: "courier new" , "courier" , monospace;">3</span></span></span></span><br />
<br />
At a public library using their wi-fi,<span style="font-family: "courier new" , "courier" , monospace;"> ifconfig </span>shows<span style="font-family: "courier new" , "courier" , monospace;"> wlan0 </span>with <span style="font-family: "courier new" , "courier" , monospace;">inet addr</span><span style="font-family: "courier new" , "courier" , monospace;"> 10.12.13.211</span>, for example, but the address changes from session to session.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"> <span style="font-family: "courier new" , "courier" , monospace;">xdebug.remote_host = 10.<span style="font-family: "courier new" , "courier" , monospace;">12.13.211</span></span></span></span><br />
<br />
This is the key difference from running in a purely local way without Docker, where<span style="font-family: "courier new" , "courier" , monospace;"> localhost </span>is a typically used value for the<span style="font-family: "courier new" , "courier" , monospace;"> remote_host </span>setting.<br />
<br />
<span style="font-size: large;">• </span>Here are the settings that need to be present in<span style="font-family: "courier new" , "courier" , monospace;"> xdebug.ini</span>. <br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> xdebug.remote_enable = On<br /> xdebug.remote_host = 10.<span style="font-family: "courier new" , "courier" , monospace;">0.0.<span style="font-family: "courier new" , "courier" , monospace;">3</span></span><br /> xdebug.remote_port = 9000</span><br />
<br />
<span style="font-size: large;">• </span>If you have trouble getting your setup to work, use a log for xDebug to record errors and warnings. Enable the log by
adding the following directive into your<span style="font-family: "courier new" , "courier" , monospace;"> xdebug.ini </span>file, e.g.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">xdebug.remote_<span style="font-family: "courier new" , "courier" , monospace;">log</span> = /tmp/xdebug.log</span></span></span><br />
<br />
This
is useful for debugging. For example, at the beginning of the
log there will probably be an indication of whether xDebug is even able to
connect to the client, which is an important clue.<br />
<br />
But use this key only as needed since it can generate a lot of log messages, some of which seem spurious.<br />
<br />
<span style="font-size: large;">• </span>For the Sublime editor, the key setting is<span style="font-family: "courier new" , "courier" , monospace;"> path_mapping</span>. Its value is an object that indicates corresponding paths. For example,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> { "/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/" }</span><br />
<br />
The key (the left side) is a path in the Docker <i>container</i> where the xDebug server finds its source code. Its value (the right side) is the path in the <i>local system</i> where Sublime finds the corresponding code.<br />
<br />
In my use case, I am only interested in the code for the Optimizely module, so I'm only providing a mapping between the two root directories of the module. <br />
<br />
<span style="font-size: large;">• </span>Here are the settings that need to be present in Sublime for its xDebug package.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> "port": 9000,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "path_mapping": {</span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> </span>"/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/"<span style="font-family: "courier new" , "courier" , monospace;"> </span>}<span style="font-family: "courier new" , "courier" , monospace;">,</span></span><br />
<br />
If you need to do troubleshooting, you might add the following setting as well in order for Sublime to output messages to its own local xDebug log.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> "debug": true</span><br />
<br />
Also use this key only as needed since it can generate a lot of log messages, some of which seem spurious.<br />
<br />
<span style="font-size: large;">• </span>No port forwarding is needed.<br />
<br />
The Docker documentation states: "By default Docker containers can make connections to the outside world ..." Since it is xDebug within the container that initiates the connection to Sublime running outside, there is no need to use port forwarding for the port between them.<br />
<br />
<span style="font-size: large;">• </span>Once you've got the correct settings for<span style="font-family: "courier new" , "courier" , monospace;"> xdebug.ini</span>, there are different ways to persist them. In my case, some settings don't change, but I work in different locations where the IP address for<span style="font-family: "courier new" , "courier" , monospace;"> remote_host </span>does vary, so I took a hybrid approach.<br />
<br />
First, I used the<span style="font-family: "courier new" , "courier" , monospace;"> docker commit </span>command to capture the current state of a container into an image. In that container I had edited<span style="font-family: "courier new" , "courier" , monospace;"> xdebug.ini </span>with the settings that remain the same.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># docker commit distracted_dijkstra drupal-xdebug</span><br />
<br />
Second, I use the<span style="font-family: "courier new" , "courier" , monospace;"> docker run </span>command with the<span style="font-family: "courier new" , "courier" , monospace;"> -e </span>option to provide the IP address as an environment variable when instantiating the image in a new container.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># docker run -e XDEBUG_CONFIG="remote_host=<span style="font-family: "courier new" , "courier" , monospace;">10.0.0.3" <span style="font-family: "courier new" , "courier" , monospace;">drupal-xdebug</span></span> </span><br />
<br />
<br />
Sources:<br />
<br />
<i>Xdebug 2 | Remote Debugging</i><br />
<a href="https://xdebug.org/docs/remote#browser_session">https://xdebug.org/docs/remote#browser_session</a><br />
<br />
<i>martomo / SublimeTextXdebug</i><br />
<a href="https://github.com/martomo/SublimeTextXdebug/blob/master/Xdebug.sublime-settings">https://github.com/martomo/SublimeTextXdebug/blob/master/Xdebug.sublime-settings</a><br />
<br />
<i>Debug your PHP in Docker with Intellij/PHPStorm and Xdebug</i><br />
<a href="https://gist.github.com/chadrien/c90927ec2d160ffea9c4">https://gist.github.com/chadrien/c90927ec2d160ffea9c4</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-39477690488965779982017-04-17T16:28:00.000-07:002017-04-17T16:28:06.153-07:00Initial Thoughts on Using DockerI have started to use the Docker images from<span style="font-family: "courier new" , "courier" , monospace;"> wadmiraal/drupal </span>for local Drupal development, for example,<span style="font-family: "courier new" , "courier" , monospace;"> wadmiraal/drupal:8.1.0 </span>to use Drupal 8.1.0. These images are very well documented at <i><a href="http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/">Use Docker to kickstart your Drupal development</a>.</i><br />
<br />
There are other images for working with Drupal, but I happened to choose this set since it incorporates the versions of PHP, MySQL, and Apache that are close to what I've been using.<br />
<br />
Here are some random notes on what I have experienced initially as someone who is new to Docker.<br />
<br />
<b>(1) Containers are easy and really fast to spin up</b><br />
(At least on my Ubuntu-based system).<br />
<br />
If
you want to start completely fresh, run a new container, which means
any changes you have made are lost. Sometimes, that's what you want, for
example, when I want a fresh install of the module I'm working on and a
clean database.<br />
<br />
On the other hand, if you want to
preserve changes to the file system of the container, you can stop it
and then start the same container later. But be aware that it's really
easy to accumulate clutter in the way of containers that you no longer
want and have to manually remove. You can see all containers, running or not, by the command:<span style="font-family: "courier new" , "courier" , monospace;"> docker ps -a</span><br />
<br />
<b>(2) There are different ways to communicate into Docker containers.</b><br />
<br />
The ways I've used are <i>port forwarding</i> and <i>volume mounting</i>.<br />
<br />
With port forwarding, when you run a container you specify which ports on the local host are passed on to a corresponding port of the container. For example, port 8080 locally can be mapped to the default port 80 of the container for http. Browsing to an address such as<span style="font-family: "courier new" , "courier" , monospace;"> localhost:8080 </span>then sends the request to the instance of the web server running in the container.<br />
<br />
Volume mounting is a way to make local directories visible to processes running in the container. I mount my local development directory for the Optimizely module to a directory path in the container's file system under the Drupal site. This allows me to edit code locally without having to do so inside the container. Nice!<br />
<br />
<b>(3) Using ssh and scp </b><br />
<br />
If you run the container with the appropriate port forwarding, you can<span style="font-family: "courier new" , "courier" , monospace;"> ssh </span>into the container. In the case of the image<span style="font-family: "courier new" , "courier" , monospace;"> wadmiraal/drupal</span>, once you have an ssh terminal the vi and vim editors are available.<br />
<br />
However, other tools that you might want are not there. These can be added on the fly by using<span style="font-family: "courier new" , "courier" , monospace;"> apt-get install</span>, for example. But keep in mind that such changes will not necessarily persist, depending on how you manage the container.<br />
<br />
If<span style="font-family: "courier new" , "courier" , monospace;"> ssh </span>is working, then so does<span style="font-family: "courier new" , "courier" , monospace;"> scp </span>for copying files back and forth between host and container. I have a one line php script that calls the function<span style="font-family: "courier new" , "courier" , monospace;"> phpinfo()</span>, which I<span style="font-family: "courier new" , "courier" , monospace;"> scp </span>from my local into the web root of the container for troubleshoot.<br />
<br />
<b>(4) Using xDebug and Sublime Text with Docker</b><br />
<br />
The Docker image has xDebug enabled for PHP, but I'd like to have xDebug running on the container to communicate with Sublime on the local system so that I can edit and step through code locally.<br />
<br />
So far, I am struggling to set this up. I expect to crack this nut eventually and will blog about it when I do.<br />
<br />
<br />
Sources:<br />
<i><br /></i>
<i>Docker Overview</i><br />
<a href="https://docs.docker.com/engine/understanding-docker/">https://docs.docker.com/engine/understanding-docker/</a> <br />
<br />
<i>Docker Tutorial Series, Part 1: An Introduction | Docker Components</i><br />
<a href="http://blog.flux7.com/blogs/docker/docker-tutorial-series-part-1-an-introduction">http://blog.flux7.com/blogs/docker/docker-tutorial-series-part-1-an-introduction</a><br />
<br />
<i>Use Docker to kickstart your Drupal development</i> <br />
<a href="http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/">http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/</a> <br />
<br />
<i>Bind container ports to the host</i><br />
<a href="https://docs.docker.com/engine/userguide/networking/default_network/binding/">https://docs.docker.com/engine/userguide/networking/default_network/binding/</a><br />
<br />
Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-80450958060972334152016-11-22T20:13:00.000-08:002016-11-22T20:13:11.307-08:00Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes)I ran into the above error message while modifying some custom Drupal code. At first, I thought I just needed to increase the memory limit due to some newly called core functions consuming more than what was allocated.<br />
<br />
There's good discussion about different ways to change the PHP memory limit in the article <i><a href="https://www.drupal.org/node/76156">Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...</a></i><br />
<br />
On my local stack, one relevant file is<span style="font-family: "courier new" , "courier" , monospace;"> /etc/php5/apache2/php.ini </span>which had a limit of 512M that I had set about two years ago. That seemed like a lot already. I upped it to 640M, but got the same error at exactly the same line of code.<br />
<br />
So I fired up a debugger to trace through. When execution reached the loop body containing the reported line where execution died (line 4 below), I single-stepped, single-stepped, single-stepped, ... and it didn't leave after the expected number of iterations.<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">1: $dirname = pathinfo($current_path, PATHINFO_DIRNAME);</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">2:</span></span> while ($dirname && $dirname != '.') {</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">3:</span></span> $page['#cache']['tags'][] = 'optimizely:' . $dirname . '/*';</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">4:</span></span> $dirname = pathinfo($dirname, PATHINFO_DIRNAME);</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">5:</span></span> }</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span></blockquote>
Line 3 kept adding entries to the<span style="font-family: "courier new" , "courier" , monospace;"> $page </span>array until memory was exhausted. It was my own coding mistake in inadvertently creating a runaway infinite loop.<br />
<span style="font-size: x-large;">∞</span><br />
<br />
Sources:<br />
<br />
<i>Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...</i><br />
<a href="https://www.drupal.org/node/76156">https://www.drupal.org/node/76156</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-40625815002249285962016-11-03T11:29:00.000-07:002016-11-03T11:29:02.995-07:00Using Drupal 8 Cache Tags for Page CachingThis is a small case study in converting how page caching is done in the D7 version of the <a href="https://www.drupal.org/project/optimizely">Optimizely</a> module to Drupal 8.<br />
<br />
The purpose of the module is to manage the insertion of certain<span style="font-family: "courier new" , "courier" , monospace;"> <script> </span>elements into designated pages of a site. To do so, the user creates one or more projects. Each project has one or more url paths. When a project is enabled, all pages that match one of its paths have the<span style="font-family: "courier new" , "courier" , monospace;"> <script> </span>element added.<br />
<br />
To specify project paths, use of a trailing * wildcard is allowed, as are special page designators. For example, these are all valid paths.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">/node/2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/node/*</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/config/system</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/*</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">*</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><front></span><br />
<br />
In the case of<span style="font-family: "courier new" , "courier" , monospace;"> /admin/* </span>it would match against any of<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/config/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/config/system/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/people</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/people/create </span><br />
...<br />
<br />
In the D7 version, invalidating is done through calls to<span style="font-family: "courier new" , "courier" , monospace;"> cache_clear_all()</span>, which takes three parameters. The function is used by the module in three different ways.<br />
<br />
(1) To invalidate a particular page, e.g.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cache_clear_all('/node/2', 'cache_page', FALSE);</span><br />
<br />
(2) To invalidate a path with a trailing wildcard, e.g.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cache_clear_all('/node/*', 'cache_page', TRUE);</span><br />
<br />
(3) To invalidate all pages of the site,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cache_clear_all('*', 'cache_page', TRUE);</span><br />
<br />
In Drupal 8 the Cache API is completely different. Function <span style="font-family: "courier new" , "courier" , monospace;">cache_clear_all() </span>has disappeared.<br />
<br />
After some research, it looked like using cache tags would be the way to go. The article <a href="https://www.drupal.org/developing/api/8/render/arrays/cacheability"><i>Cacheability of render arrays</i></a> was especially helpful in how to think about caching as applied to page rendering<i>.</i><br />
<br />
There are two facets to implementing this. The first is what needs to be done when a page is rendered, the second is what to invalidate when triggering changes occur.<br />
<br />
For page rendering, I already had an implementation of<span style="font-family: "courier new" , "courier" , monospace;"> hook_page_attachments() </span>that checked for inserting the element into any page whose path matched against any of the enabled project paths.<br />
<br />
This hook function is where cache tags could be added to the page. This turned out to be a little tricky. A page must be invalidated for two different use cases: when the page contains the element which now needs to be removed, and when the page does not contain the element but it now needs to be added.<br />
<br />
For the first case, I decided to just use the matching project path act as the cache tag (there can only be one because overlapping project paths are not allowed).<br />
<br />
But for the second case, I had to cover all the possible project paths that <i>might be enabled</i> in the future. So, for example, for the page at<span style="font-family: "courier new" , "courier" , monospace;"> /node/2 </span>there are three such possible paths.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> /node/2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> /node/*</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> *</span><br />
<br />
For every page rendered, it was necessary to attach all of these possible project paths as cache tags. Here is a snippet of code that shows how this is done in<span style="font-family: "courier new" , "courier" , monospace;"> hook_page_attachments()</span>.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> // Site-wide wildcard.</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> $page['#cache']['tags'][] = 'optimizely:*';<br /><br /> // Non site-wide wildcards. Repeat for every directory level.<br /> $dirname = pathinfo($current_path, PATHINFO_DIRNAME);<br /> while ($dirname && $dirname != '/') {<br /> $page['#cache']['tags'][] = 'optimizely:' . $dirname . '/*';<br /> $dirname = pathinfo($dirname, PATHINFO_DIRNAME);<br /> }<br /><br /> // The specific page url.<br /> $page['#cache']['tags'][] = 'optimizely:' . $current_path;</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><br /> // Finally, if there is an alias for the page, tag it.<br /> if ($current_path_alias) {<br /> $page['#cache']['tags'][] = 'optimizely:' . $current_path_alias;<br /> } </span></span><br />
<br />
The<span style="font-family: "courier new" , "courier" , monospace;"> optimizely: </span>prefix follows the convention of prefixing a group name as part of the tag where appropriate. For example, cache tags from core include<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">node:1</span> </span>and<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">config:node.type.article</span></span>.<br />
<br />
Finally, there is the matter of what and how to invalidate when changes to the projects and their paths are submitted. This turned out to be fairly easy to implement.<br />
<br />
An array of all relevant project paths is passed to a function that carries out the following.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> $cache_tags = [];<br /> foreach ($path_array as $path) {<br /> $cache_tags[] = 'optimizely:' . $path;<br /> }<br /> </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> \Drupal::service('cache_tags.invalidator')->invalidateTags($cache_tags);</span></span><br />
<br />
<hr width="85%" />
<br />
For debugging purposes, outputting<span style="font-family: "courier new" , "courier" , monospace;"> X-Drupal-Cache-Tags </span>in HTTP headers was extremely useful. See my earlier post <i><a href="https://optimizely-to-drupal-8.blogspot.com/2016/10/enable-and-use-x-drupal-cache-tags-in.html">Enable and Use X-Drupal-Cache-Tags in HTTP headers</a></i>. <br />
<br />
This post is about the caching that is done by Drupal itself. The D7 version also checks for the presence of the<span style="font-family: "courier new" , "courier" , monospace;"> varnish </span>module and calls a function of that module if it exists. I did not pursue a replacement for that functionality, but the articles <a href="https://www.drupal.org/developing/api/8/cache/tags/varnish"><i>Varnish</i></a> and <a href="http://www.jeffgeerling.com/blog/2016/use-drupal-8-cache-tags-varnish-and-purge"><i>Use Drupal 8 Cache Tags with Varnish and Purge</i></a> look promising.<br />
<br />
<br />
Sources:<br />
<br />
<i>Function cache_clear_all() has been removed</i> <br />
<a href="https://optimizely-to-drupal-8.blogspot.com/2014/07/function-cacheclearall-has-been-removed.html">https://optimizely-to-drupal-8.blogspot.com/2014/07/function-cacheclearall-has-been-removed.html</a><br />
<br />
<i>Cacheability of render arrays</i><br />
<a href="https://www.drupal.org/developing/api/8/render/arrays/cacheability">https://www.drupal.org/developing/api/8/render/arrays/cacheability</a><br />
<br />
<i>Cache tags</i> <br />
<a href="https://www.drupal.org/developing/api/8/cache/tags">https://www.drupal.org/developing/api/8/cache/tags</a><br />
<br />
<i>Allow to set #cache metadata in hook_page_attachments()</i> <a href="https://www.drupal.org/node/2475749">https://www.drupal.org/node/2475749</a><br />
<br />
<i>public static function Cache::invalidateTags</i><br />
<a href="https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Cache%21Cache.php/function/Cache%3A%3AinvalidateTags/8">https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Cache!Cache.php/function/Cache%3A%3AinvalidateTags/8</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-52385714736151313292016-10-17T20:41:00.000-07:002016-10-17T20:41:28.476-07:00Enable and Use X-Drupal-Cache-Tags in HTTP headersWhile converting from Drupal 7's caching to that of Drupal 8, for debugging purposes I wanted to enable the display of <span style="font-family: "courier new" , "courier" , monospace;">X-Drupal-Cache-Tags</span> in HTTP headers.<br />
<br />
How to do this is documented in the article <a href="https://www.drupal.org/developing/api/8/response/cacheable-response-interface"><i>CacheableResponseInterface</i></a> and is actually simple, but I had enough trouble with getting this to work that I'm providing a few of my own notes here.<br />
<br />
In Drupal 8 core, there is the file<span style="font-family: "courier new" , "courier" , monospace;"> /sites/default/default.services.yml </span> Copy this file to the same directory, creating a new file named<span style="font-family: "courier new" , "courier" , monospace;"> services.yml </span>(assuming you don't already have such a file).<br />
<br />
For our purposes, in<span style="font-family: "courier new" , "courier" , monospace;"> services.yml </span>the only key you need is <span style="font-family: "courier new" , "courier" , monospace;">http.response.debug_cacheability_headers</span>.<br />
<br />
In my situation, I ended up with a<span style="font-family: "courier new" , "courier" , monospace;"> services.yml </span>that only contained the following two lines, where the value of the key is changed to<span style="font-family: "courier new" , "courier" , monospace;"> true</span>. I deleted all of the other keys.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> parameters:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> http.response.debug_cacheability_headers: true</span><br />
<br />
After creating and/or editing<span style="font-family: "courier new" , "courier" , monospace;"> services.yml </span><b><i>be sure</i></b> to do "a container rebuild, which is necessary when changing a container parameter". One way to accomplish this is via<span style="font-family: "courier new" , "courier" , monospace;"> Clear all caches </span>in the admin UI.<br />
<br />
Finally, to see the display of <span style="font-family: "courier new" , "courier" , monospace;"> X-Drupal-Cache-Tags </span>in HTTP headers, I use the Chrome browser on a Linux system. It happens to be version 39.0. <br />
<br />
Navigate to:<span style="font-family: "courier new" , "courier" , monospace;"> More tools > Developer tools > Network tab</span><br />
<br />
After loading a page whose cache tags you are interested in, on the left side of the<span style="font-family: "courier new" , "courier" , monospace;"> Developer tools </span>panel, click on the url for the page. Then on the right, click on the<span style="font-family: "courier new" , "courier" , monospace;"> Headers </span>subtab and look under<span style="font-family: "courier new" , "courier" , monospace;"> Response Headers </span>for the<span style="font-family: "courier new" , "courier" , monospace;"> X-Drupal-Cache-Tags </span>property. You will see a number of cache tags that come with Drupal core in addition to your own custom ones.<br />
<br />
Source: <br />
<br />
<i>CacheableResponseInterface</i><br />
<a href="https://www.drupal.org/developing/api/8/response/cacheable-response-interface#debugging">https://www.drupal.org/developing/api/8/response/cacheable-response-interface#debugging</a><br />
Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-34458782072874933302016-08-30T23:10:00.001-07:002016-08-30T23:10:29.247-07:00Call of MySQL database function failsThis post describes (another) database issue that was tracked down by my colleague Luis Delacruz.<br />
<br />
We had both a live site and a corresponding development site using the same MySQL database server. On the live site, carrying out a particular user task worked fine, but doing the same task on the development site would fail with the message "<span style="font-family: "courier new" , "courier" , monospace;">DB Error: unknown error</span>". <br />
<br />
The problem was eventually traced back to the fact that the contents of the database for the development site had been loaded by importing a backup of the live site.<br />
<br />
The live site contained several database functions. For example, here's one of the exported function definitions.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> CREATE DEFINER=`user1`@`localhost` </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> FUNCTION `clean_string`(in_str varchar(4096)) </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> RETURNS varchar(4096) CHARSET latin1<br /><br /> BEGIN <br /> /** <br /> * Function will strip all non-ASCII and unwanted ASCII characters in string <br /> * <br /> * @author Shay Anderson 10.11 <br /> * <br /> * @param VARCHAR in_arg <br /> * @return VARCHAR <br /> */ <br /> DECLARE i, len SMALLINT DEFAULT 1; <br /> DECLARE ret CHAR(255) DEFAULT ''; <br /> DECLARE c CHAR(1); <br /> SET len = CHAR_LENGTH( in_str ); <br /> REPEAT <br /> BEGIN <br /> SET c = MID( in_str, i, 1 ); <br /> IF c REGEXP '[[:alnum:]]' THEN <br /> SET ret=CONCAT(ret,c); <br /> END IF; <br /> SET i = i + 1; <br /> END; <br /> UNTIL i > len END REPEAT; <br /> RETURN ret; <br /> END</span></span><br />
<br />
Luis noticed that the definitions of this database function as well as others included the clause <span style="font-family: "courier new" , "courier" , monospace;">DEFINER = 'user1'@'localhost'</span>.<br />
<br />
By default, the<span style="font-family: "courier new" , "courier" , monospace;"> SQL SECURITY </span>characteristic of a function is<span style="font-family: "courier new" , "courier" , monospace;"> DEFINER</span>. That means that when the routine is executed, it does so within the <i>security context</i> of the user specified as the<span style="font-family: "courier new" , "courier" , monospace;"> DEFINER</span>.<br />
<br />
This worked fine for the live site because its database runs under user <span style="font-family: "courier new" , "courier" , monospace;">user1</span>.<br />
<br />
However, for the staging site, that user does not have access to the staging database, so it was denied. That is, the user <i>did not have sufficient permissions for executing the function body as applied to the other database</i>.<br />
<br />
Deleting the<span style="font-family: "courier new" , "courier" , monospace;"> DEFINER </span>clause and reloading the function confirmed that the clause was the problem.<br />
<br />
Also, a key factor was that both databases are managed by the same db server, within which<span style="font-family: "courier new" , "courier" , monospace;"> user1 </span>exists. If the function definition were imported into a different db server for which<span style="font-family: "Courier New",Courier,monospace;"> user1 </span>does not exist, it could be that the problem would not occur.<br />
<br />
Further research uncovered this advice:<br />
<br />
"For a stored routine or view, use<span style="font-family: "Courier New",Courier,monospace;"> SQL SECURITY INVOKER </span>in the object definition when possible so that it can be used only by users with permissions appropriate for the operations performed by the object. ".<br />
<br />
In other words, set the<span style="font-family: "courier new" , "courier" , monospace;"> SQL SECURITY </span>characteristic to<span style="font-family: "Courier New",Courier,monospace;"> INVOKER </span> so that the security context of whichever user invokes the function is in effect. This seems ideal.<br />
<br />
So we modified the beginning part of the function definition as follows:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> CREATE <b><strike>DEFINER=`user1`@`localhost`</strike></b> </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> FUNCTION `clean_string`(in_str varchar(4096)) </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> RETURNS varchar(4096) CHARSET latin1<br /> <b>SQL SECURITY INVOKER</b><br /> </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> BEGIN </span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> ....</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> END</span></span><br />
<br />
<br />
(Incidentally, the error message "<span style="font-family: "courier new" , "courier" , monospace;">DB Error: unknown error</span>" is apparently output by the Drupal 7 CiviCRM module.)<br />
<br />
Sources:<br />
<br />
<i>Access Control for Stored Programs and Views</i><br />
<a href="https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html">https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html</a><br />
<br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-19369774469411243662016-05-28T15:24:00.000-07:002016-05-28T15:24:02.805-07:00Two sites on the same database server - PDOException: SQLSTATE[HY000][1129] Host 'nnn.nnn.nnn.nnn' is blocked because of many connection errorsA web search shows a number of forums and postings about the error message in the title of this post, but our particular problem involved two sites on the same web server and the same database server. One of the sites appeared to repeatedly bring down the other site.<br />
<br />
We had had a working live site implemented in Drupal and wanted to add a separate staging site in order to have a more transparent workflow that is less risky.<br />
<br />
So for the staging site, a separate Drupal instance was installed and a separate Drupal database was created using the same respective servers as for the live site.<br />
<br />
The sites use the CiviCRM module, which has its own<span style="font-family: "courier new" , "courier" , monospace;"> civicrm.settings.php </span>file for configuration.<br />
<br />
- - - - -<br />
<br />
After staging was created, testing of the staging site was showing very odd errors in the form of inconsistent user profiles and such.<br />
<br />
With help from the primary developers, I was able to track down that the <span style="font-family: "courier new" , "courier" , monospace;">civicrm.settings.php </span>file was incorrectly specifying the live database for use by the staging site. Instead, it should have been specifying the staging database.<br />
<br />
I corrected the configuration in<span style="font-family: "courier new" , "courier" , monospace;"> civicrm.settings.php </span>so that staging was accessing its own database via its own database user.<br />
<br />
Here are the original constants defined in<span style="font-family: "courier new" , "courier" , monospace;"> civicrm.settings.php</span> <br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">// These ip addresses are not the actual ones.<br />define( 'CIVICRM_UF_DSN', 'mysql://live_db_user:user_password@</span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">12.210.138.193</span></span>/live_db?new_link=true' ); <br /><br />define( 'CIVICRM_DSN', 'mysql://live_db_user:user_password@</span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">12.210.138.193</span></span>/live_db?new_link=true' ); </span></span><br />
<br />
And the revised constants in<span style="font-family: "courier new" , "courier" , monospace;"> civicrm.settings.php</span> <br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">// These ip addresses are not the actual ones.<br />define( 'CIVICRM_UF_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' ); <br /><br />define( 'CIVICRM_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' );</span></span><br />
<br />
I then emailed our tester about the changes late at night, without doing any of my own testing. Mea culpa.<br />
<br />
That's when things got interesting.<br />
<br />
- - - - - <br />
<br />
The next morning, <i>both</i> the staging and the live sites were broken with the same error even though I had not changed any code on live.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">PDOException: SQLSTATE[HY000][1129] Host '12.210.138.193' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' in lock_may_be_available() (line 167 of /var/www/oursite.org/httpdocs/includes/lock.inc).</span> </span><br />
<br />
Since the error message stated "<span style="font-family: "courier new" , "courier" , monospace;">unblock with 'mysqladmin flush-hosts' </span>", I logged onto the database server and did<span style="font-family: "courier new" , "courier" , monospace;"> flush-hosts </span>on the live database in order to keep the live site running. Live came back up. And I sighed with relief thinking that was enough.<br />
<br />
Then I browsed to the staging site again to troubleshoot it. Now it came up with a different error.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">PDOException: SQLSTATE[28000] [1045] Access denied for user 'staging_db_us<span style="font-family: "courier new" , "courier" , monospace;">er</span>'@'12.210.138.193' (using password: YES) in lock_may_be_available() (line 167 of /var/www/staging.oursite.org/httpdocs/includes/lock.inc). </span></span><br />
<br />
A bit later, I browsed to the live site again. And it was broken <i>again</i>.<br />
<br />
The pattern was: (1) fix the live site by running the<span style="font-family: "courier new" , "courier" , monospace;"> flush-hosts </span>command, (2) browse to the staging site, (3) and the live site is broken again.<br />
<br />
Since keeping the live site running was a priority, this was driving me crazy! <br />
<br />
Those of you who are more familiar with MySQL than I was may already know what had happened. With hindsight, the issue is so obvious that it makes me reflect on my own thinking processes.<br />
<br />
- - - - - <br />
<br />
It was my colleague Luis who cracked this. The following explanation is due to him.<br />
<br />
The basic problem is that the database user<span style="font-family: "courier new" , "courier" , monospace;"> staging_db_user </span>did not have privileges to access the database via the<span style="font-family: "courier new" , "courier" , monospace;"> 12.210.138.193 </span>host address (although it did have access through a different ip). So whenever I browsed to the staging site, connection errors would occur. <br />
<br />
The number of connection errors would quickly accumulate and exceed the allowed maximum. The relevant variable in MySQL is<span style="font-family: "courier new" , "courier" , monospace;"> max_connect_errors</span>, with a default value of something like 1000.<br />
<br />
When the maximum number of errors was reached, the MySQL server would then block <i>any</i> further connection requests from that host ip, including for the live site!<br />
<br />
<i>This explains why browsing to the staging site would result in access errors for the live site as well. Because of the common ip, the two sites were inadvertently coupled and mutually dependent.</i><br />
<br />
In the end, we decided to use a different host ip address for the staging site to access the database precisely to avoid this cross-site brittleness.<br />
<br />
Source:<br />
<br />
<i>Troubleshooting Problems Connecting to MySQL</i><br />
<a href="https://dev.mysql.com/doc/refman/5.7/en/problems-connecting.html">https://dev.mysql.com/doc/refman/5.7/en/problems-connecting.html</a><br />
<br />
<br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-64937998135724989202016-03-15T07:30:00.001-07:002016-03-15T07:30:58.069-07:00Blocking Chinese and Korean spam for user-submitted content I help maintain a Drupal 7 site called <a href="http://www.powerpoetry.org/">Power Poetry</a> that is a platform for publishing poetry. It's really a great site with awesome writing that has grown steadily in both the amount of content and the number of page views.<br />
<br />
Unfortunately, its popularity has apparently attracted some foreign spam content, particularly in Chinese and Korean.<br />
<br />
Our initial take was that being U.S.-centric, we should limit submissions to only English and Spanish. So I did some research into language detection, found a service with an API that looked promising, and did an initial implementation. However, for technical reasons that I never did uncover, that API was not working on our hosting setup (even though it worked on my local system!).<br />
<br />
We then changed our focus from language detection to the fact that our current problem was almost entirely due to submissions made in certain <i>scripts</i> (specific human writing systems).<br />
<br />
I found that regular expressions in PHP support <a href="https://secure.php.net/manual/en/regexp.reference.unicode.php">Unicode character properties</a>. Those property codes include values that designate the Chinese script (which includes both traditional and simplified characters) and the Korean script.<br />
<br />
We decided to take a zero-tolerance approach. A single Chinese or Korean character causes the user submission not to validate. So far, implementing this has drastically reduced the amount of inappropriate content and comments.<br />
<br />
Here's the code, which is called by our Drupal form validation functions.<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">/**<br /> * Check whether a string contains any characters from a</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"> * banned script, such as Chinese or Korean.<br /> *<br /> * @param string $text<br /> * The piece of text to be checked.<br /> *<br /> * @return TRUE | FALSE<br /> * Returns TRUE if any Chinese or Korean characters detected.<br /> * Otherwise, returns FALSE.<br /> */<br />function _contains_banned_scripts($text) {<br /><br /> $unicode_modifier = 'u';<br /><br /> // Detect whether there are any Chinese characters.<br /> $chinese_regex = '\p{Han}+';<br /> $preg_regex = '/' . $chinese_regex . '/' . $unicode_modifier;<br /> $chinese_found = preg_match($preg_regex, $text);<br /><br /> if ($chinese_found == 1) {<br /> return TRUE;<br /> }<br /><br /> // Detect whether there are any Korean characters.<br /> $korean_regex = '\p{Hangul}+';<br /> $preg_regex = '/' . $korean_regex . '/' . $unicode_modifier;<br /> $korean_found = preg_match($preg_regex, $text);<br /><br /> if ($korean_found == 1) {<br /> return TRUE;<br /> }<br /><br /> return FALSE;<br />}</span></span><br />
<br />
Sources:<br />
<br />
<i>Forum spam</i><br />
<a href="https://en.wikipedia.org/wiki/Forum_spam">https://en.wikipedia.org/wiki/Forum_spam</a><br />
<br />
<i>Unicode Regular Expressions</i><br />
<a href="http://www.regular-expressions.info/unicode.html">http://www.regular-expressions.info/unicode.html</a> <br />
<br />
<i>Unicode Character Properties</i><br />
<a href="https://secure.php.net/manual/en/regexp.reference.unicode.php">https://secure.php.net/manual/en/regexp.reference.unicode.php</a>Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-81876921690861998442016-02-24T18:20:00.000-08:002019-08-26T17:50:54.998-07:00An Exercise in Pair ProgrammingI've spent many years as a software developer working in contented solitude: wrapping my mind around spaghetti code; stepping through a debugger; implementing an algorithm. I'm an introvert, so I'm comfortable being alone.<br />
<br />
The past few years, though, I've felt pulled to collaborate more with others and to participate more actively as a member of a team. I've done this by communicating with end users, clients, and non-technical team members as well as doing informal mentoring and teaching.<br />
<br />
I've known about <i><a href="http://guide.agilealliance.org/guide/pairing.html">pair programming</a></i> for some time but have never practiced it. So when it turned out that one of my colleagues and I were both learning jQuery and JavaScript, I suggested we do an exercise together via pair programming.<br />
<br />
Luis and I have pretty different backgrounds. He is a college student majoring in computer engineering. I'm a very seasoned programmer who later moved into web technologies and have re-entered that field after a long hiatus.<br />
<br />
We built a client-side game for a person to scramble and then solve the <a href="https://en.wikipedia.org/wiki/15_puzzle">15-puzzle</a> (source code in <a href="https://github.com/tz-earl/15-Puzzle">GitHub repo</a> ). This was done over a few weeks during off hours when we were able to schedule snippets of time together.<br />
<br />
(<b>Update</b>: <a href="https://tz-earl.github.io/15-Puzzle/">play the puzzle</a> )<br />
<br />
It went really well, and I want to jot down a few thoughts about what enabled it to be a positive, constructive experience.<br />
<br />
* <b>We already had a good working relationship</b>.<br />
<br />
Luis and I are both part-time developers at a <a href="http://www.wholewhale.com/">digital consulting agency</a>. We had worked together before and felt personally compatible, so much so that we've made a point of going into the office on the same days so that we can readily consult with each other.<br />
<br />
* <b>We did it as part of a learning exercise</b>.<br />
<br />
This was done not as part of our paid work but of our individual learning endeavors. Luis has been using one of the online services, I've been going through a traditional book. This meant that not much was at stake in terms of producing results, which reduced the potential stress level considerably. <br />
<br />
* <b>Neither of us dominated</b>.<br />
<br />
Although I am far senior in terms of general software experience, I am a novice with respect to JavaScript and jQuery. And most recently, my career path has been a pretty choppy one with a lot of non-technical roads taken and committed to. These factors together with a strong preference to collaborate rather than compete means that I simply treat Luis as a peer.<br />
<br />
* <b>We were open to each other's criticism and suggestions</b>.<br />
<br />
If you can sufficiently suspend your own preferences, habits, and beliefs about what's "right", it's possible to learn a lot from another developer. Each of us was willing to try coding differently. This ran the gamut from how to do indentation to what data structures to use for the underlying model.<br />
<br />
Small example: in jQuery there is consistent and frequent use of <i>anonymous</i> functions that are passed as parameters. Luis suggested giving names to those functions even though the names are never used. Although this appears to go against the most common coding practice, I ended up liking the additional clarity and expression of intent that such <i>unnecessary</i> function names provide.<br />
<br />
* <b>Debugging went much faster and more efficiently</b>.<br />
<br />
Case in point. Luis realized that certain event handlers were not working because they no longer existed -- the DOM elements they were attached to were being removed and then some of those elements were re-created as replacements, minus their handlers. Stuff like that is obvious when someone else points it out!<br />
<br />
For my part, I correctly suspected that a recursive use of a particular jQuery selector was causing the failure of an iterator to affect all intended objects. It's a subtle bug that I still don't understand the root cause of, but my background in programming languages led to a good guess.<br />
<br />
* <b>Both of us have an academic background</b>.<br />
<br />
This allowed me, for example, to jump into explaining terminology such as "operator precedence" and "operator associativity" without Luis' eyes glazing over. He not only has the intellectual chops, he gets it that taking the time to learn basic concepts and acquire fundamental understanding is worth it.<br />
<br />
Doing this exercise has given me momentum and encouragement to look for opportunities to do more pair programming and to expand further my own envelope. Nice!<br />
<br />
Resources:<br />
<br />
<i>Pair Programming</i><br />
<a href="http://guide.agilealliance.org/guide/pairing.html">http://guide.agilealliance.org/guide/pairing.html</a> <br />
<br />
<i>Pair Programming Considered Harmful?</i><br />
<a href="http://techcrunch.com/2012/03/03/pair-programming-considered-harmful/">http://techcrunch.com/2012/03/03/pair-programming-considered-harmful/</a> <br />
<br />
<i>Pair Programming vs. Code Reviews</i><br />
<a href="https://blog.codinghorror.com/pair-programming-vs-code-reviews/">https://blog.codinghorror.com/pair-programming-vs-code-reviews/</a>Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-85472331490088554462016-02-01T08:08:00.000-08:002016-02-01T08:08:40.691-08:00Unwanted Side Effect of Reducing Permissions on Drupal Text FormatsOn a Drupal 7 site, we were getting malicious comments that were triggering undesired page redirects, as I wrote about in <i><a href="http://optimizely-to-drupal-8.blogspot.com/2015/11/malicious-page-redirects-and-drupals.html">Malicious page redirects and Drupal's Filtered-HTML text format</a></i>.<br />
<br />
One cause of those redirects was malicious html using the<span style="font-family: "courier new" , "courier" , monospace;"> <meta> </span>tag. This tag was allowed because<span style="font-family: "courier new" , "courier" , monospace;"> Authenticated Users </span>had access to the<span style="font-family: "courier new" , "courier" , monospace;"> Full HTML </span>text format. So we decided to limit use of<span style="font-family: "courier new" , "courier" , monospace;"> Full HTML </span>only to<span style="font-family: "courier new" , "courier" , monospace;"> Administrators</span>.<br />
<br />
Since then, I have learned that this caused an undesirable side-effect. The symptom was that when a user tried to edit content that they had previously created, in the text area for doing the editing they got the following message.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">This field has been disabled because you do not have sufficient permissions to edit it</span><br />
<br />
Very puzzling! Why would a user not be able to edit their own content that they had created?<br />
<br />
It turns out that when text content is created, the text format in effect at that time is <i>stored with the content and is applied when editing is done later</i>.<br />
<br />
If you limit the roles that have access to a particular text format, then text content created earlier by users who have that role may no longer be editable by those users. They are blocked from making any changes.<br />
<br />
On our site, this is not a serious problem. Text content is almost always written and submitted once and not revised further.<br />
<br />
<span style="font-size: xx-small;"> * * * </span><br />
<span style="font-size: xx-small;"><br /></span>
However, a site with content that is actively edited is going to run into a wall. There are a couple of things you might be able to do.<br />
<br />
If the number of content items is small, then while logged in as an Administrator or some other role with sufficient permissions, edit each item so that it uses a Text Format that the author has permission to use. In our case, it would have been from<span style="font-family: "courier new" , "courier" , monospace;"> Full HTML </span>to<span style="font-family: "courier new" , "courier" , monospace;"> Filtered HTML</span>.<br />
<br />
If you have a lot of content and you're dangerous enough to hack around directly with the database, then a database global change will also work.<br />
<br />
For example, in our case, we have two tables that are relevant. Executing these SQL commands changes the values.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> update field_data_body </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> set body_format = 'filtered_html' </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> where body_format = 'full_html';</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> update field_revision_body</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> set body_format = 'filtered_html' </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> where body_format = 'full_html';</span><br />
<br />
(And don't forget to clear cache!)<br />
<br />
Thanks to David Needham for a particularly <a href="https://www.drupal.org/node/1064168#comment-7692605">pithy and helpful comment</a> in the second article cited below.<br />
<br />
Sources:<br />
<br />
<i>Signature box for authenticated users - This field has been disabled because you do not have sufficient permissions to edit it </i><br />
<a href="https://www.drupal.org/node/1034064">https://www.drupal.org/node/1034064</a><br />
<br />
<i>This field has been disabled because you do not have sufficient permissions to edit it</i><br />
<a href="https://www.drupal.org/node/1064168">https://www.drupal.org/node/1064168</a>Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-37019911021534170852016-01-15T14:55:00.000-08:002016-01-28T14:09:02.223-08:00jQuery.get() Ajax call fails on localhost urlWhile going through the book <i><a href="https://www.packtpub.com/web-development/learning-jquery-fourth-edition">Learning jQuery, 4th Edition</a></i>, I've been doing the exercises by placing and editing code files in a local directory on the desktop and opening<span style="font-family: "courier new" , "courier" , monospace;"> index.html </span>as a file in Firefox.<br />
<br />
This worked fine until I had to code an Ajax call that referred to a php file. This required the use of a web server, so I created a subdirectory under the local server root and copied the php file into the subdir.<br />
<br />
Keeping all other code on the desktop, I then edited the Ajax call so that it requested from the url <br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> http://localhost/learning-jquery/e.php</span><br />
<br />
The call silently failed and returned nothing. Console windows would show nothing, no errors.<br />
<br />
Eventually, I found out about the <b><a href="https://en.wikipedia.org/wiki/Same-origin_policy">same-origin policy</a></b>. In brief, this policy means that by default JavaScript is prevented from making requests across domain boundaries, where same-origin means that the protocol, hostname, and port number must be identical.<br />
<br />
So I moved all code files into the subdir under the server root. Working from there rather than the desktop, the Ajax calls then succeeded.<br />
<br />
(And I changed the requested url back to just<span style="font-family: "Courier New",Courier,monospace;"> e.php</span>) <br />
<br />
***<br />
<br />
But what about carrying out
cross-origin access? One possible way is to configure the server to
allow it. For Apache on my Linux system, I added the
following directive in<span style="font-family: "courier new" , "courier" , monospace;"> /etc/apache2/apache2.conf</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> <Directory /var/www/html/learning-jquery><br /> Header set Access-Control-Allow-Origin "*"<br /> </Directory></span><br />
<br />
However, checking this configuration change with the command<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> # apachectl -t</span><br />
<br />
resulted in<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">AH00526: Syntax error on line 205 of /etc/apache2/apache2.conf:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">Action '-t' failed.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">The Apache error log may have more information.</span><br />
<br />
It turns out that I also had to enable a module named<span style="font-family: "courier new" , "courier" , monospace;"> headers</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> # a2enmod headers</span><br />
<br />
Then restart the server. That enabled cross-origin access.<br />
<br />
Not yet satisfied, I was wondering if the wildcard<span style="font-family: "courier new" , "courier" , monospace;"> "*" </span>in
the directive could be replaced by something more restrictive. For the
way I'm developing locally, the following also works because opening a
file via the<span style="font-family: "courier new" , "courier" , monospace;"> file:// </span>protocol results in an origin of<span style="font-family: "courier new" , "courier" , monospace;"> null</span>.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> <Directory /var/www/html/learning-jquery><br /> Header set Access-Control-Allow-Origin "null"<br /> </Directory></span><br />
<br />
***<br />
<span style="font-size: xx-small;"> </span><br />
The fact that the calls had failed silently was very troubling. One way to avoid this is to register a global Ajax error handler by calling<span style="font-family: "courier new" , "courier" , monospace;"> ajaxError()</span>. Here's a handler that simply puts up an alert box. (I'm using jQuery 1.9)<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">$(document).ready(<br /> function setAjaxErrorHandler() {<br /> <b>$(document).ajaxError</b>(<br /> function alertError(event, jqxhr, settings, thrownErr) {<br /> alert('Ajax error handler: ' + jqxhr.status </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> + ' ' + jqxhr.statusText);<br /> }<br /> ); <br /> }<br />);</span><br />
<br />
For the cross-origin error, this puts up the message "<span style="font-family: "courier new" , "courier" , monospace;">Ajax error handler: 0 error</span>", which is not exactly super helpful but better than nothing. <br />
<br />
Another way is to attach an error handler to the particular request by using the<span style="font-family: "courier new" , "courier" , monospace;"> fail() </span>method. For example,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">$.get('http://localhost/learning-jquery/e.php', { ... }, </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> function (data) { ... })</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> .<b>fail</b>(function (jqxhr) {<br /> alert('GET error: ' + jqxhr.status + ' ' </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;"> + jqxhr.statusText);<br /> });</span><br />
<br />
<br />
Sources:<br />
<br />
<i>jQuery.get()</i><br />
<a href="https://api.jquery.com/jQuery.get/">https://api.jquery.com/jQuery.get/</a><br />
<br />
<i>Same-origin policy</i><br />
<a href="https://en.wikipedia.org/wiki/Same-origin_policy">https://en.wikipedia.org/wiki/Same-origin_policy</a><br />
<br />
<i>Why is CORS important?</i><br />
<a href="http://enable-cors.org/">http://enable-cors.org/</a><br />
<br />
<i>CORS on Apache</i><br />
<a href="http://enable-cors.org/server_apache.html">http://enable-cors.org/server_apache.html</a><br />
<br />
<i>Configure Apache To Accept Cross-Site XMLHttpRequests on Ubuntu</i><br />
<a href="https://harthur.wordpress.com/2009/10/15/configure-apache-to-accept-cross-site-xmlhttprequests-on-ubuntu/">https://harthur.wordpress.com/2009/10/15/configure-apache-to-accept-cross-site-xmlhttprequests-on-ubuntu/</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com2tag:blogger.com,1999:blog-5023448542823447711.post-56242674795023473892016-01-07T09:52:00.000-08:002016-01-07T09:52:12.870-08:00How to Extract Columns from a CSV file (without using a spreadsheet)One of my colleagues who does data analysis had a CSV (comma-separated values) file that he could not open in Excel nor in any other spreadsheet program he tried.<br />
<br />
Either due to its sheer filesize of 125MB, or its number of rows of more than 1,500,000, the programs would gag.<br />
<br />
It turned out that for his purposes, he did not need <i>all</i> of the data, only a subset of the columns. Maybe extracting only what he needed into a smaller CSV would enable him to be able to work with it.<br />
<br />
I had earlier read parts of the book <i><a href="http://linuxcommand.org/tlcl.php">The Linux Command Line</a></i>, by William Shotts. I vaguely remembered mention of a utility to selectively pull fields out of a text file. That turned out to be the<span style="font-family: "Courier New",Courier,monospace;"> cut </span>command.<br />
<br />
Here's what the first few rows of the original data file looked like.<br />
<br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">id,type,distance,userid,charityID,time,lat,lon</span></span></span><br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">1003529743,walk,4.48342,1000545086,2166731,"2015-06-30 00:00:05",40.2501,-76.6714<br />1003529744,Run,4.087,1000402641,15048,"2015-06-30 00:00:21",45.5244,-89.7398<br />1003529745,run,2.631135,1000258381,61635018,"2015-06-30 00:00:23",41.6281,-87.193<br />1003529746,Bike,1.216703,1000505816,18010,"2015-06-30 00:00:24",43.0306,-78.7963<br />1003529747,walk,2.069664,1000015957,18010,"2015-06-30 00:00:25",39.2481,-76.5165<br />1003529748,Bike,6.174,1000126350,18010,"2015-06-30 00:00:25",29.5913,-82.4298<br />1003529749,run,1.47652,1000542869,92985044,"2015-06-30 00:00:26",40.7115,-89.4287</span></span></span><br />
<br />
Only the<span style="font-family: "Courier New",Courier,monospace;"> type </span>and<span style="font-family: "Courier New",Courier,monospace;"> userid </span>columns were actually required. Using the following command I was able to generate another CSV with just those two fields.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ cut -f 2,4 -d, july.csv > type-userid.csv</span><br />
<br />
The<span style="font-family: "Courier New",Courier,monospace;"> -f </span>option specifies which fields to extract. In this case, its the 2nd and the 4th fields. The<span style="font-family: "Courier New",Courier,monospace;"> -d </span>option is the field delimiting character, which in our case is the comma. It defaults to tabs.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">july.csv </span>is the input file,<span style="font-family: "Courier New",Courier,monospace;"> type-userid.csv </span>captures the standard-out.<br />
<br />
The first few rows of the resulting file were<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">type,userid</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">walk,1000545086<br />Run,1000402641<br />run,1000258381<br />Bike,1000505816<br />walk,1000015957<br />Bike,1000126350<br />run,1000542869</span></span><br />
<br />
And the filesize was reduced to about 24MB, which was usable and much more manageable.<br />
<br />
(However, depending on which spreadsheet app you are using, the row count might exceed the limit. For example, both Excel 2007 and LibreOffice 4.2 Calc can handle a maximum of 1,048,576 rows.)<br />
<br />
*** <br />
<br />
The<span style="font-family: "Courier New",Courier,monospace;"> cut </span>command works blazingly fast. It only took about a second to process the input file. After I handed off the outputted file, I later realized that I might have been able to save my colleague some tedium and waiting time by applying other CLI text-processing commands as well, such as<span style="font-family: "Courier New",Courier,monospace;"> sort</span>,<span style="font-family: "Courier New",Courier,monospace;"> uniq</span>, and<span style="font-family: "Courier New",Courier,monospace;"> wc</span>, depending on what he wanted to do.<br />
<br />
Source:<br />
<br />
<i>The Linux Command Line</i>, by William Shotts<br />
<a href="http://linuxcommand.org/tlcl.php">http://linuxcommand.org/tlcl.php</a><br />
(You can buy the paper book or download the pdf for free)<br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-14186476520080158892015-12-26T07:25:00.001-08:002015-12-26T07:25:18.979-08:00Drupal 8, Beta 15 --> RC 1: Eliminating use of checkPlain( )Since<span style="font-family: "courier new" , "courier" , monospace;"> SafeMarkup::checkPlain() </span>is now deprecated and possibly intended to be removed along with other<span style="font-family: "courier new" , "courier" , monospace;"> SafeMarkup </span>methods, I decided to address this in migrating to RC 1.<br />
<br />
In the common use case where D8's Twig templates are used for output, you can dispense with<span style="font-family: "courier new" , "courier" , monospace;"> checkPlain() </span>entirely and rely on <a href="https://www.drupal.org/node/2296163#overview">Twig's autoescaping</a>. That turned out to apply to our simple use of forms to input and render a few text values.<br />
<br />
In the original code,<span style="font-family: "courier new" , "courier" , monospace;"> checkPlain() </span>was called both to process user-entered values as well as to sanitize those values before they were placed into render arrays. When I typed<span style="font-family: "courier new" , "courier" , monospace;"> <script> </span>into a form field, it was incorrectly escaped twice and displayed as <span style="font-family: "courier new" , "courier" , monospace;">&lt;script&gt; </span><br />
<br />
Removing those calls works fine and conforms to what is considered good practice for D8. Potentially unsafe markup is stored as is in the database, but Twig converts it before it is sent to the browser.<br />
<br />
The article <a href="https://www.drupal.org/node/2549395"><i>SafeMarkup methods are removed</i></a> is extremely useful in how it breaks down the different use cases for<span style="font-family: "courier new" , "courier" , monospace;"> checkPlain() </span>and what needs to be done differently for each in D8 in order to prevent unsafe markup from being rendered.<br />
<br />
Besides Twig templates, the other use cases are:<br />
<br />
- Text placed into a render array by using the<span style="font-family: "Courier New",Courier,monospace;"> #plain_text </span>key<br />
- A mixture of escaped markup with markup not to be escaped<br />
- Non-HTML responses, eg. JSON<br />
<br />
The
discussion also applies to the check_plain() function in Drupal 7, for which<span style="font-family: "Courier New",Courier,monospace;"> checkPlain() </span>was a replacement. If
you're starting out with a conversion from D7, the article is a must
read.<br />
<br />
Sources:<br />
<br />
<i>SafeMarkup::set(), SafeMarkup::checkPlain(), and other methods are removed from Drupal 8 core</i><br />
<a href="https://groups.drupal.org/node/478558">https://groups.drupal.org/node/478558</a><br />
<br />
<i>SafeMarkup methods are removed</i><br />
<a href="https://www.drupal.org/node/2549395#check_plain">https://www.drupal.org/node/2549395</a><br />
<br />
<i>Twig autoescape enabled and text sanitization APIs updated</i><br />
<a href="https://www.drupal.org/node/2296163">https://www.drupal.org/node/2296163</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-57711369565269572292015-12-19T07:15:00.000-08:002015-12-19T07:15:43.067-08:00Module: "Configuration inspector for Drupal 8"In my recent post on <i><a href="http://optimizely-to-drupal-8.blogspot.com/2015/12/drupal-8-beta-14-beta-15-missing.html">Missing langcode in configuration schema</a></i> I discovered that the<span style="font-family: "courier new" , "courier" , monospace;"> langcode </span>key in configuration schema is now required by default by<span style="font-family: "courier new" , "courier" , monospace;"> Testing </span>module automated tests.<br />
<br />
The test results had error messages that said "<span style="font-family: "Courier New",Courier,monospace;">Uncaught PHP Exception Drupal\Core\Config\Schema\SchemaIncompleteException</span>" and "<span style="font-family: "Courier New",Courier,monospace;">langcode missing schema</span>". <br />
<br />
As a result of that debugging, I got interested in a module called <span style="font-family: "courier new" , "courier" , monospace;">Configuration Inspector for Drupal 8</span>.<br />
<br />
Installing the module adds a new tab named<span style="font-family: "courier new" , "courier" , monospace;"> Inspect </span>to the page at <span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/admin/config/development/configuration</span><br />
<br />
That tab shows all of the<span style="font-family: "courier new" , "courier" , monospace;"> Configuration Keys </span>on the site, to which core contributes quite a few.<br />
<br />
For the settings for our module, after I removed the<span style="font-family: "courier new" , "courier" , monospace;"> langcode </span>key from the<span style="font-family: "courier new" , "courier" , monospace;"> .schema.yml </span>file, the tab showed that the configuration key had<span style="font-family: "courier new" , "courier" , monospace;"> 1 error</span>. Going to the<span style="font-family: "courier new" , "courier" , monospace;"> Raw Data </span>page for the key then showed under<span style="font-family: "courier new" , "courier" , monospace;"> Configuration Validation</span>,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">array ( </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 'optimizely.settings:langcode' => 'missing schema',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">)</span><br />
<br />
This is the gist of what the exceptions had said in the results from running the automated tests, but if I had used this config inspector early on to <i>validate the schema</i>, I would have spotted the error sooner rather than after the testing failed.<br />
<br />
<span style="font-size: x-small;"> * * * </span><br />
<br />
The inspector also shows the types and the values of individual items. In our case, this includes a project id number whose value had been submitted through a form and stored programmatically by using the <a href="https://www.drupal.org/node/1809490">Simple Configuration API</a>.<br />
<br />
There was no entry shown for the<span style="font-family: "courier new" , "courier" , monospace;"> langcode </span>item, which is defined but had no value. I was able to provide one by adding code in<span style="font-family: "courier new" , "courier" , monospace;"> hook_install()</span>, again, by using the Simple Configuration API.<br />
<br />
Sources:<br />
<br />
<i>Configuration inspector for Drupal 8</i><br />
<a href="https://www.drupal.org/project/config_inspector">https://www.drupal.org/project/config_inspector</a><br />
<br />
<i>Configuration API in Drupal 8</i><br />
<a href="https://www.drupal.org/node/1667894">https://www.drupal.org/node/1667894</a><br />
<br />
<i>Configuration schema/metadata</i><br />
<a href="https://www.drupal.org/node/1905070">https://www.drupal.org/node/1905070</a><br />
Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com2tag:blogger.com,1999:blog-5023448542823447711.post-77444640689805032852015-12-12T21:49:00.000-08:002015-12-26T07:18:17.965-08:00Drupal 8, Beta 14 --> Beta 15: Missing langcode in configuration schemaIn migrating from Drupal 8 beta 14 to beta 15, the functionality of the module itself worked fine, but some of the automated tests were failing with error messages that included the following.<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Uncaught PHP Exception
Drupal\Core\Config\Schema\SchemaIncompleteException: "Schema errors for
optimizely.settings with the following errors:
optimizely.settings:langcode missing schema" at <span style="font-family: "courier new" , "courier" , monospace;">/var/www/html/opti</span>/core/lib/Drupal/Core/Config/Testing/ConfigSchemaChecker.php
line 98
</span></blockquote>
In this message,<span style="font-family: "courier new" , "courier" , monospace;"> optimizely.settings </span>is the name of a group of configuration settings. It is also a configuration key in a<span style="font-family: "courier new" , "courier" , monospace;"> .schema.yml </span>file that is required for automated testing, which I blogged about earlier at <a href="https://www.blogger.com/"><i><span id="goog_549828910"></span></i></a><i><a href="http://optimizely-to-drupal-8.blogspot.com/2014/12/beta-3-beta-4-configuration-schema-and.html"><span id="goog_549828919"></span>Beta 3 --> Beta 4: Configuration schema and metadata</a></i><a href="https://www.blogger.com/"><span id="goog_549828911"></span></a><span id="goog_549828920"></span>.<br />
<br />
The article <i><a href="https://www.drupal.org/node/2547365">Fix config schema</a></i> mentioned a very similar error message and stated "Because of a recent core change all tests are failing". I looked at the patches in that article, but I could not figure out what needed to be added in our case.<br />
<br />
Then a search through the core code for the string "<span style="font-family: "courier new" , "courier" , monospace;">langcode:</span>" led me to add<span style="font-family: "courier new" , "courier" , monospace;"> langcode: </span>as a key to the<span style="font-family: "courier new" , "courier" , monospace;"> .schema.yml </span>file as in the following.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">optimizely.settings:<br /> type: mapping<br /> label: 'Optimizely Config Data'<br /> mapping:<br /> optimizely_id:<br /> type: integer<br /> label: 'Optimizely ID Number'<br /> translatable: false<br /><b> langcode:<br /> type: string<br /> label: 'Language code'</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
Problem solved. All automated tests passed after this change.<br />
<br />
<b>Update</b>: Also see my later post on <i><a href="http://optimizely-to-drupal-8.blogspot.com/2015/12/module-configuration-inspector-for.html">Module: Configuration Inspector for Drupal 8</a></i>. <br />
<br />
Sources:<br />
<br />
<i>Beta 3 --> Beta 4: Configuration schema and metadata</i><br />
<a href="http://optimizely-to-drupal-8.blogspot.com/2014/12/beta-3-beta-4-configuration-schema-and.html">http://optimizely-to-drupal-8.blogspot.com/2014/12/beta-3-beta-4-configuration-schema-and.html</a><br />
<br />
<i>Fix config schema</i><br />
<a href="https://www.drupal.org/node/2547365">https://www.drupal.org/node/2547365</a><br />
<br />
<i>All TestBase derived tests now enforce strict configuration schema adherence by default</i><br />
<a href="https://www.drupal.org/node/2391795">https://www.drupal.org/node/2391795</a><br />
<br />
<i>Configuration schema/metadata</i><br />
<a href="https://www.drupal.org/node/1905070">https://www.drupal.org/node/1905070</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com1tag:blogger.com,1999:blog-5023448542823447711.post-12925775318548486902015-11-10T08:11:00.000-08:002015-11-10T08:11:52.491-08:00"Notice: Undefined index: und in eval() (line . . . . . /modules/php/php.module(80) : eval()'d code)"On my local instance of a Drupal 7 site, I'd get warnings like the following on almost every page for a custom content type called<span style="font-family: "courier new" , "courier" , monospace;"> Poem</span>.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Notice: Undefined index: und in eval() (line 12 of /var/www/html/power-poetry/modules/php/php.module(80) : eval()'d code). </span><br />
<br />
Although most probably innocuous, these warnings were also being logged numerous times in the system log, cluttering it up and making it harder to spot other, significant messages. It was enough of an annoyance that I decided to fix it.<br />
<br />
I was able to track this to a snippet of PHP code that is used in a Drupal block of ours. The code is part of the block's<span style="font-family: "courier new" , "courier" , monospace;"> Visibility Settings</span>. It determines if the current<span style="font-family: "courier new" , "courier" , monospace;"> Poem </span>being rendered satisfies certain criteria or not. If so, the block is made visible.<br />
<br />
The offending line turned out to be<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> $slam_id = $current_slam['und'][0]['target_id'];</span><br />
<br />
The variable<span style="font-family: "courier new" , "courier" , monospace;"> $current_slam </span>was often an empty array. So by replacing the line with an additional check, the PHP warnings disappeared.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> if (!empty($current_slam)) { </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> $slam_id = $current_slam['und'][0]['target_id']; </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }</span><br />
<br />
The key to finding this quickly was remembering that we have this <i>user-supplied snippet of code that needs to be evaluated at runtime</i>. <br />
<br />
- - - - -<br />
<br />
While debugging, I wanted to see the type and the value of the variable<span style="font-family: "courier new" , "courier" , monospace;"> $current_slam</span>. The following worked to output it to the system log.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> watchdog('debug', print_r($current_slam, TRUE), array(), </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> WATCHDOG_NOTICE);</span><br />
<br />
The second parameter to<span style="font-family: "courier new" , "courier" , monospace;"> print_r() </span>determines what kind of return value the function passes back. By default,<span style="font-family: "courier new" , "courier" , monospace;"> print_r() </span>returns its status. But by setting the second param to<span style="font-family: "courier new" , "courier" , monospace;"> TRUE</span>, the return value is the output string instead.<br />
<br />
- - - - -<br />
<br />
The site is hosted on Pantheon, which provides this status message: <br />
<br />
<ul class="check-list" style="-webkit-tap-highlight-color: transparent; -webkit-text-stroke-width: 0px; background-color: #fefbec; box-sizing: border-box; color: #595959; font-family: 'Open Sans', Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; list-style: none; margin: 0px 0px 10px; orphans: auto; padding: 0px 0px 0px 3em; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<li class="severity-fail" style="-webkit-tap-highlight-color: transparent; box-sizing: border-box;"><div class="result" style="-webkit-tap-highlight-color: transparent; box-sizing: border-box; line-height: 1.5; margin: 0px !important;">
<b style="-webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #ff004f; font-weight: bold;">PHP Filter:</b><span class="Apple-converted-space"> </span>PHP Filter is enabled! Executable code should never be stored in the database, and support for this feature was removed in Drupal 8 - https://drupal.org/node/1203886</div>
<div class="action" style="-webkit-tap-highlight-color: transparent; box-sizing: border-box; font-style: italic; line-height: 1.5; margin: 0px !important;">
Remove all executable code from your content and move it to your codebase.</div>
</li>
</ul>
<br />
Source:<br />
<br />
<i>Remove the PHP module from Drupal core</i><br />
<a href="https://drupal.org/node/1203886">https://drupal.org/node/1203886</a> <br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com5tag:blogger.com,1999:blog-5023448542823447711.post-61171101715718443112015-11-03T22:26:00.000-08:002015-11-03T22:26:17.751-08:00Malicious page redirects and Drupal's Filtered-HTML text formatOne of the Drupal 7 sites I maintain is<span style="font-family: "courier new" , "courier" , monospace;"> <a href="http://www.powerpoetry.org/">www.powerpoetry.org</a> </span>which is a platform for publishing poems and comments about them.<br />
<br />
Recently, we heard from an irate poet. When browsing to one of her poems, the page would partially load, then automatically redirect to a movie streaming site that had nothing to do with her poem nor with the site in general. Her poem page somehow got hijacked, and readers could never read her work.<br />
<br />
It turned out that all site users, even Anonymous ones, had permissions for the<span style="font-family: "courier new" , "courier" , monospace;"> Full HTML Text Format</span>. This allowed comments to be submitted that contained malicious code to redirect the page.<br />
<br />
So far, I have found two different techniques that were used to implement redirection.<br />
<br />
(1) Use of the<span style="font-family: "courier new" , "courier" , monospace;"> <meta> </span>tag with an attribute of<span style="font-family: "courier new" , "courier" , monospace;"> http-equiv="refresh"</span>. For example,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <meta http-equiv="refresh" </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> content="2;url=http://scumbags.com/"></span><br />
<br />
where the value of 2 means a delay of two seconds before the refresh.<br />
<br />
(2) Use of the<span style="font-family: "courier new" , "courier" , monospace;"> onmouseover </span>attribute to execute JavaScript. For example,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><a href="//tinyurl.com/nm8ug<span style="font-family: "courier new" , "courier" , monospace;">h</span>" </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> onmouseover="document.location='//tinyurl.com/nm8ug<span style="font-family: "courier new" , "courier" , monospace;">h</span>';"></span><br />
<br />
As documented, this attribute and closely related ones are valid for almost all HTML tags, which would make it potentially a big problem where markup is allowed.<br />
<br />
If you have good knowledge of HTML, these techniques may be obvious, but for me as a back-end developer they were new. <br />
<br />
Because I have direct access to the Drupal database, I was able to use a SQL query such as the following to find malicious comments.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> select entity_id, comment_body_value</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> from field_data_comment_body</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> where comment_body_value like "%onmouseover%";</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> </span>- - - - -<br />
<br />
To help plug these security holes, we decided to allow only Administrators to have access to all<span style="font-family: "courier new" , "courier" , monospace;"> Text Formats</span>. Depending on the needs of your site, you might want to disable the<span style="font-family: "courier new" , "courier" , monospace;"> Full HTML </span>format entirely.<br />
<br />
For all other users, the Filtered HTML Text Format was permitted, configured using<span style="font-family: "courier new" , "courier" , monospace;"> Limit Allowed HTML Tags </span>so that only the following tags are processed.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <p> <br> <em> <strong> <cite> <blockquote> <ul> <ol> <li></span><br />
<br />
Although not part of this security problem, we purposely disallowed<span style="font-family: "courier new" , "courier" , monospace;"> <a> </span>tags, deciding that they are not essential for comments on our site and almost always only appear in spam comments. As part of eliminating links, we also turned off<span style="font-family: "courier new" , "courier" , monospace;"> Convert URLs Into Links</span>. <br />
<br />
Note that the user can still type in whatever they want, including any kind of HTML. The original text is stored unchanged as the content of the comment.<br />
<br />
However, what the<span style="font-family: "courier new" , "courier" , monospace;"> Filtered HTML </span>format does is effectively strip out disallowed tags as part of the rendering process.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> </span>- - - - -<br />
<br />
Even with<span style="font-family: "courier new" , "courier" , monospace;"> Filtered HTML</span>, I was concerned that the<span style="font-family: "courier new" , "courier" , monospace;"> onmouseover </span>attribute could still be used inside one of the tags that is allowed. What about something like<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <p onmouseover="...."></span><br />
<br />
But some testing and some stepping through core code revealed that<span style="font-family: "courier new" , "courier" , monospace;"> onmouseover </span>as well as other risky attributes are stripped out of those tags for output even though the tags themselves are processed.<br />
<br />
The holes were plugged for these two exploits.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> </span>- - - - -<br />
<br />
The final thing left to do was to go back and check again the malicious comments, which I had left in the database while carrying out the above steps. I thought that applying<span style="font-family: "courier new" , "courier" , monospace;"> Filtered HTML </span>would neutralize them. I was wrong.<br />
<br />
In the database table<span style="font-family: "courier new" , "courier" , monospace;"> field_data_comment_body </span>there is a column<span style="font-family: "courier new" , "courier" , monospace;"> comment_body_format </span>that stores the format in effect <i>when the comment was created</i>. The format apparently is used to render the comment even if it is no longer allowed.<br />
<br />
As a result, those comments were still redirecting!<br />
<br />
So I used SQL queries to find their entity ids, then deleted them by browsing to, for example, the following path where 30259 is the entity id of a comment.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> /comment/30259/edit</span><br />
<br />
or<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> /comment/30259/delete</span><br />
<br />
The latter path goes directly to the delete confirmation dialog. In some cases I had to use it because loading the edit form for the comment triggered the redirection, so <i>I couldn't even edit it</i>.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> </span>- - - - -<br />
<br />
Michael Richardson at <a href="https://pantheon.io/">Pantheon</a> support did an awesome job of doing the initial troubleshooting when I was at a loss as to where to start. It was he who uncovered the use of<span style="font-family: "courier new" , "courier" , monospace;"> onmouseover </span>on our site. Thanks, Michael!<br />
<br />
<br />
Sources:<br />
<br />
<i>What is the Meta Refresh Tag?</i><br />
<a href="http://webdesign.about.com/od/metataglibraries/a/aa080300a.htm">http://webdesign.about.com/od/metataglibraries/a/aa080300a.htm</a> <br />
<i><br /></i>
<i>HTML onmouseover Event Attribute</i><br />
<a href="http://www.w3schools.com/tags/ev_onmouseover.asp">http://www.w3schools.com/tags/ev_onmouseover.asp</a><br />
<br />
<i>Drupal Text Formats and Filters Tutorial</i> <br />
<a href="https://www.hostknox.com/tutorials/drupal/formats-and-filters">https://www.hostknox.com/tutorials/drupal/formats-and-filters</a><br />
<br />
<i>Text Filters and Input Formats</i> <br />
<a href="https://www.drupal.org/node/213156">https://www.drupal.org/node/213156</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0tag:blogger.com,1999:blog-5023448542823447711.post-77463140076245316602015-09-21T09:41:00.000-07:002016-11-12T17:29:19.531-08:00Shareable URLs from Forward and the Twilio service: accessing an API endpoint on a localhost siteHow do you have an external service access an API endpoint that is defined on a site that you're running locally? <br />
<br />
One of the Drupal 7 sites (<a href="http://allgoodtext.com/">AllGoodText.com</a>) I work on sends out SMS messages to users who register for weekly messages to be delivered to their cellphones.<br />
<br />
The site uses the <a href="https://www.twilio.com/">Twilio</a> service to configure a virtual cellphone number from which the messages are apparently sent.<br />
<br />
We also use the D7 <a href="https://www.drupal.org/project/twilio">Twilio module</a> (version 7.x-1.10) to interface with the Twilio API. The module provides Drupal-specific elements such as hooks for implementing the desired functionality.<br />
<br />
During local development, this worked fine as long as the site was only calling the Twilio API endpoints.<br />
<br />
However, we wanted to enhance the site to define its own endpoints for the Twilio service to call back. For example, we wanted to be notified of and to be able to process STOP messages that our users were texting to our virtual cell number so that we could update the site's database.<br />
<br />
For development purposes, I wanted to test as realistically as possible. I wanted my local site to interact with my cellphone via Twilio.<br />
<br />
The problem was, I only run Apache on my system as a <i>local server</i>. At first, I thought I'd have to use our development site (it happens to be on Pantheon), constantly pushing code changes to it. That would have been tedious and messy.<br />
<br />
Because I find the use of a debugger indispensable, I also would have had to figure out how to do so remotely.<br />
<br />
Luckily, I saw a <a href="http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh">comment</a> in response to a question about how to test the use of Twilio that mentioned "request forwarding utilities", of which <a href="https://forwardhq.com/">Forward</a> is one. The comment is clear and concise, so I'm just going to quote the commenter, Devin Rader.<br />
<blockquote class="tr_bq">
Basically, request forwarding utilities like ForwardHQ do two things:<br />
<ol>
<li>It creates a publicly accessible URL, that you can tell Twilio about. </li>
<li>It creates an SSH tunnel between a port on your machine and a server on the internet</li>
</ol>
When Twilio needs to make an HTTP request because it received an inbound SMS or voice call, it will request the ForwardHQ URL. Forward knows how to take that request and send it to the port running on your local machine. That port maps to your local web development server running in order to process the request and return the result to Twilio.</blockquote>
<br />
This sounded like exactly what I wanted. Here are the steps I took.<br />
<br />
1. Signed up for an account with <a href="https://forwardhq.com/">Forward</a>.<br />
<br />
2. Installed their browser extension for Chrome, which puts an icon on the toolbar to<span style="font-family: "courier new" , "courier" , monospace;"> Open Forward</span>.<br />
<br />
3. Browsed to the local site. Clicked on the<span style="font-family: "courier new" , "courier" , monospace;"> Open Forward </span>icon.<br />
<br />
(I probably had to log in to Forward the first time I did this.)<br />
<br />
4. This brought up a special page from the browser extension. Clicked on<span style="font-family: "courier new" , "courier" , monospace;"> Start Tunnel </span>to create a tunnel between the site running on my localhost and Forward.<br />
<br />
To access the site, the url<span style="font-family: "courier new" , "courier" , monospace;"> wholewhale.fwd.wf </span>was provided by default, where the subdomain<span style="font-family: "courier new" , "courier" , monospace;"> wholewhale </span>was specified by me.<br />
<br />
5. The Twilio module defines the path<span style="font-family: "courier new" , "courier" , monospace;"> /twilio/sms </span>using<span style="font-family: "courier new" , "courier" , monospace;"> hook_menu() </span>for processing incoming sms messages.<br />
<br />
6. So under my Twilio account, I specified<span style="font-family: "courier new" , "courier" , monospace;"> http://wholewhale.fwd.wf/twilio/sms </span>as the<span style="font-family: "courier new" , "courier" , monospace;"> Request URL </span>to notify the local site of incoming sms messages from users.<br />
<br />
(On the live site, the endpoint will be <span style="font-family: "courier new" , "courier" , monospace;"> http://allgoodtext.com/twilio/sms<span style="font-family: inherit;">)</span></span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: inherit;"> </span></span><br />
<span style="font-family: inherit;"><br /></span>
7. I had an issue with the Twilio module logging the error message "<span style="font-family: "courier new" , "courier" , monospace;">Incoming SMS could not be validated</span>" and not invoking the hook I had implemented.<br />
<br />
I hacked around this by temporarily inserting a "<span style="font-family: "courier new" , "courier" , monospace;">return TRUE</span>" statement in the module code so that it would continue as though no error had occurred. This worked fine for what I was doing.<br />
<br />
I was now able to run the site and make code changes to it locally, run my debugger locally, and have the site interact with the Twilio service in both directions.<br />
<br />
Sweet!!<br />
<br />
<br />
Sources:<br />
<br />
<i>Forward</i><br />
<a href="https://forwardhq.com/">https://forwardhq.com/</a><br />
<br />
<i>twilio</i> (the service)<br />
<a href="https://www.twilio.com/">https://www.twilio.com/</a><br />
<br />
<i>Twilio</i> (the Drupal module)<br />
<a href="https://www.drupal.org/project/twilio">https://www.drupal.org/project/twilio</a><br />
<br />
<i>How to test Twilio calling my REST API via POST when a user sends an sms to a short code?</i><br />
<a href="http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh">http://stackoverflow.com/questions/17983398/how-to-test-twilio-calling-my-rest-api-via-post-when-a-user-sends-an-sms-to-a-sh</a><br />
<br />
<i>Accessing localhost from Anywhere</i><br />
<a href="http://www.sitepoint.com/accessing-localhost-from-anywhere/">http://www.sitepoint.com/accessing-localhost-from-anywhere/</a><br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com1tag:blogger.com,1999:blog-5023448542823447711.post-63399085193265502502015-08-24T17:50:00.001-07:002015-08-24T17:50:59.090-07:00Beta 11 --> Beta 12: _format property in route definitionMigrating to Beta 12, some ajax functionality was not functioning as expected. The log messages reported an exception,<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException: No route found for the specified format</span><br />
<br />
Using a debugger to step through the function that throws the exception, I made guesses as to what was needed. Eventually, I stumbled on a fix.<br />
<br />
In the<span style="font-family: "Courier New",Courier,monospace;"> .routing.yml </span>file I changed the relevant route definition by removing the<span style="font-family: "Courier New",Courier,monospace;"> _format </span>property as follows.<br />
<br />
<b>Before</b>:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">ajax.enable:<br /> path: /ajax/optimizely<br /> defaults:<br /> _controller: \Drupal\optimizely\AjaxEnable::enableDisable<br /> _title: Optimizely Administer AJAX<br /> requirements:<br /> _permission: administer optimizely<br /><span style="color: red;"> <b>_format: json</b></span></span><br />
<br />
<b>After</b>:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">ajax.enable:<br /> path: /ajax/optimizely<br /> defaults:<br /> _controller: \Drupal\optimizely\AjaxEnable::enableDisable<br /> _title: Optimizely Administer AJAX<br /> requirements:<br /> _permission: administer optimizely</span><br />
<br />
Unfortunately, searching through the Drupal and the Symfony docs, I have not found anything to help me understand <i>why</i> this change to the route definition makes a critical difference.<br />
<br />
From the module's point of view, it looks as though the property is actually not needed. The client-side JavaScript that makes the request to the server sets<span style="font-family: "Courier New",Courier,monospace;"> json </span>as the<span style="font-family: "Courier New",Courier,monospace;"> dataType</span>. And the server-side code always instantiates a<span style="font-family: "Courier New",Courier,monospace;"> JsonResponse </span>object to pass back.<br />
<br />
<br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com1tag:blogger.com,1999:blog-5023448542823447711.post-23771457197817854832015-08-19T08:27:00.000-07:002015-08-24T17:35:19.822-07:00Beta 11 --> Beta 12: Leading slash required in paths for "Add Alias" page (and elsewhere)The<span style="font-family: "Courier New",Courier,monospace;"> Add Alias </span>page that can be accessed via<span style="font-family: "Courier New",Courier,monospace;"> admin/config/search/path/add </span>now requires that values entered for both of the fields<span style="font-family: "Courier New",Courier,monospace;"> Existing System Path </span>and<span style="font-family: "Courier New",Courier,monospace;"> Path Alias </span>must start with a leading slash. Otherwise, the form does not validate.<br />
<br />
When filling in the form manually, error messages are displayed to that effect, so it's clear what the problem is.<br />
<br />
However, when this is being done programmatically in an automated<span style="font-family: "Courier New",Courier,monospace;"> Testing </span>case, it's not at all obvious what went wrong.<br />
<br />
Here's a piece of original code to create a path alias to an existing node, written for a subclass of<span style="font-family: "Courier New",Courier,monospace;"> WebTestBase</span>:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit = array();</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit['source'] = 'node/' . $node->id();</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit['alias'] = $this->randomMachineName(10);</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));</span></span><br />
<br />
And here's the corrected code:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit = array();</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit['source'] = '<b><span style="color: red;">/</span></b>node/' . $node->id();</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $edit['alias'] = <span style="color: red;"><b>'/' . </b></span>$this->randomMachineName(10);</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> $this->drupalPostForm($this->addAliasPage, $edit, t('Save'));</span></span><br />
<br />
<hr width="95%" />
<br />
There seems to be an overall pattern that paths relative to the site root must now start with a slash. Besides these fields for the<span style="font-family: "Courier New",Courier,monospace;"> Add Alias </span>page, I wrote in <a href="http://optimizely-to-drupal-8.blogspot.com/2015/08/beta-11-beta-12-aliasmanager-expects.html">a previous post</a> about such a requirement in calls to some methods of class<span style="font-family: "Courier New",Courier,monospace;"> AliasManager</span>. And there's recent mention of this in <i><a href="https://www.drupal.org/node/2550615">Update the DB dump to have a leading slash for the frontpage</a></i>.<br />
<br />
<br />
<br />Earl Fonghttp://www.blogger.com/profile/05682372868411855409noreply@blogger.com0