Using the code from Creating a configuration entity type in Drupal 8 to create a sample module for demonstrating use of configuration entities, I got the following log message when browsing to the path /admin/config/system/example
Drupal\Component\Plugin\Exception\PluginNotFoundException: The "example" entity type does not exist. in Drupal\Core\Entity\EntityTypeManager->getDefinition() (line 133 of /var/www/html/opti/core/lib/Drupal/Core/Entity/EntityTypeManager.php).
This is embarrassing, but I'm going to write it up anyway.
I had copied and pasted the code verbatim from the article into new files using the filenames and paths that were indicated. It should have worked!
It turns out I had forgotten one little thing: at the top of each of the php files, I had neglected to insert
<?php
What eventually tipped me off was that my text editor, Sublime Text, was not providing the syntax coloring that it normally did.
Laughing my ass off ...
Sources:
Creating a configuration entity type in Drupal 8
https://www.drupal.org/docs/8/api/configuration-api/creating-a-configuration-entity-type-in-drupal-8
Configuration Entities in Drupal 8
https://wunder.io/blog/configuration-entities-in-drupal-8/2014-07-14
Sep 2, 2017
How to enable Vagrant SSH access into a Docker container
Vagrant seems to work really well with VirtualBox and some other virtualization providers, but I had a situation where I wanted to get it to run with Docker instead (on my local Linux Mint system) and to enable Vagrant to have ssh access into the Docker container.
I started with the Docker image wadmiraal/drupal:7.54 which I had been using for doing Drupal development via Docker alone. There are several changes that had to be made, some in the Docker image, some in the Vagrantfile, in order to satisfy Vagrant expectations.
(1) Create a user named vagrant in the guest system.
Running the image in a Docker container, I ssh'ed into it using its existing root user, then created a new user "vagrant". Set the password to "vagrant", which is a convention but it could be a different password.
I tested this new user by logging in manually.
# ssh -p 2222 vagrant@127.0.0.1
where 2222 is the local host port that is forwarded to the ssh port on the guest.
(2) Provide ssh key authentication in the guest system.
On the host / local system, I already had ssh keys. I used the following command to set up the key in the guest system.
# ssh-copy-id -p 2222 vagrant@127.0.0.1
Again, I tested this by logging in manually.
(3) Enable sudo for the vagrant user in the guest system.
The Vagrant docs say, "Many aspects of Vagrant expect the default SSH user to have passwordless sudo configured. This lets Vagrant configure networks, mount synced folders, install software, and more."
The Docker image I started with did not even have sudo available, so I logged into the guest as root and installed it.
# apt-get update
# apt-get install sudo
Then used visudo to edit /etc/sudoers to insert two lines.
# visudo
vagrant ALL=(ALL) NOPASSWD:ALL
Defaults:vagrant !requiretty
After making these changes to the guest, I used Docker to commit a new image. Let's call it earl/drupal:7.54
(4) Add Docker settings in the Vagrantfile.
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'docker'
Vagrant.configure("2") do |config|
config.vm.network "forwarded_port", guest: 22, host: 2222
config.vm.provider "docker" do |dock|
dock.image = "earl/drupal:7.54"
dock.name = "drupal-7.54"
dock.has_ssh = true
end
end
Specifying the default provider in the Vagrantfile is just a convenience so that you don't have to use the --provider option for the vagrant up command.
(5) Set up password authentication for ssh in the Vagrantfile.
Vagrant.configure("2") do |config|
config.ssh.username = "vagrant"
config.ssh.password = "vagrant"
end
This is optional. If you provide config.ssh.password as above, Vagrant will use password authentication. Otherwise, Vagrant will default to key authentication, as in the following step.
(6) Set up key authentication for ssh in the Vagrantfile.
Vagrant.configure("2") do |config|
config.ssh.keys_only = false
config.ssh.private_key_path = "/home/earl/.ssh/id_rsa"
end
This is also optional. Do either step (5) or step (6). If you do both, Vagrant will use password authentication.
ssh.keys_only must be set to false in order to use your own ssh keys, and you must also provide the path to those keys.
(7) Test that Vagrant is able to make a change in the guest.
For example, I added the following two lines to the Vagrantfile to execute a shell command.
Vagrant.configure("2") do |config|
config.vm.provision "shell",
inline: "touch /vagrant/hello-world"
end
Doing vagrant up should boot up without errors and execute the shell command successfully. In the above snippet, because Vagrant automatically synchronizes the /vagrant directory on the guest with the host directory where the Vagrantfile is located, you can check for the hello-world file on the host.
Sources:
Creating a Base Box
https://www.vagrantup.com/docs/boxes/base.html
SSH Settings
https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html
Docker Configuration
https://www.vagrantup.com/docs/docker/configuration.html
I started with the Docker image wadmiraal/drupal:7.54 which I had been using for doing Drupal development via Docker alone. There are several changes that had to be made, some in the Docker image, some in the Vagrantfile, in order to satisfy Vagrant expectations.
(1) Create a user named vagrant in the guest system.
Running the image in a Docker container, I ssh'ed into it using its existing root user, then created a new user "vagrant". Set the password to "vagrant", which is a convention but it could be a different password.
I tested this new user by logging in manually.
# ssh -p 2222 vagrant@127.0.0.1
where 2222 is the local host port that is forwarded to the ssh port on the guest.
(2) Provide ssh key authentication in the guest system.
On the host / local system, I already had ssh keys. I used the following command to set up the key in the guest system.
# ssh-copy-id -p 2222 vagrant@127.0.0.1
Again, I tested this by logging in manually.
(3) Enable sudo for the vagrant user in the guest system.
The Vagrant docs say, "Many aspects of Vagrant expect the default SSH user to have passwordless sudo configured. This lets Vagrant configure networks, mount synced folders, install software, and more."
The Docker image I started with did not even have sudo available, so I logged into the guest as root and installed it.
# apt-get update
# apt-get install sudo
Then used visudo to edit /etc/sudoers to insert two lines.
# visudo
vagrant ALL=(ALL) NOPASSWD:ALL
Defaults:vagrant !requiretty
After making these changes to the guest, I used Docker to commit a new image. Let's call it earl/drupal:7.54
(4) Add Docker settings in the Vagrantfile.
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'docker'
Vagrant.configure("2") do |config|
config.vm.network "forwarded_port", guest: 22, host: 2222
config.vm.provider "docker" do |dock|
dock.image = "earl/drupal:7.54"
dock.name = "drupal-7.54"
dock.has_ssh = true
end
end
Specifying the default provider in the Vagrantfile is just a convenience so that you don't have to use the --provider option for the vagrant up command.
(5) Set up password authentication for ssh in the Vagrantfile.
Vagrant.configure("2") do |config|
config.ssh.username = "vagrant"
config.ssh.password = "vagrant"
end
This is optional. If you provide config.ssh.password as above, Vagrant will use password authentication. Otherwise, Vagrant will default to key authentication, as in the following step.
(6) Set up key authentication for ssh in the Vagrantfile.
Vagrant.configure("2") do |config|
config.ssh.keys_only = false
config.ssh.private_key_path = "/home/earl/.ssh/id_rsa"
end
This is also optional. Do either step (5) or step (6). If you do both, Vagrant will use password authentication.
ssh.keys_only must be set to false in order to use your own ssh keys, and you must also provide the path to those keys.
(7) Test that Vagrant is able to make a change in the guest.
For example, I added the following two lines to the Vagrantfile to execute a shell command.
Vagrant.configure("2") do |config|
config.vm.provision "shell",
inline: "touch /vagrant/hello-world"
end
Doing vagrant up should boot up without errors and execute the shell command successfully. In the above snippet, because Vagrant automatically synchronizes the /vagrant directory on the guest with the host directory where the Vagrantfile is located, you can check for the hello-world file on the host.
Sources:
Creating a Base Box
https://www.vagrantup.com/docs/boxes/base.html
SSH Settings
https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html
Docker Configuration
https://www.vagrantup.com/docs/docker/configuration.html
Aug 25, 2017
Moving Docker's storage to a different location
On a Linux Mint system, I was running low on disk space in the partition where Docker CE 17.06 had installed itself and was storing its files such as images. So I thought I'd move the docker storage directory to a different partition that had a lot more space, and create a symbolic link in the directory's original location to where it had been moved.
That sounded pretty simple, but it turned out to be a vexing problem.
After moving the directory, I ran a Docker container that contained an instance of Drupal 7. When I browsed to the Drupal site I got the error:
I had typed in several cp commands before running the container again, but I didn't have a clear memory or record of the steps I had taken. After the first occurrence of the error, I did further copy commands. The error kept occurring, but once in a while the site came up okay. This thrashing about included my re-initializing the docker directory at least once.
From my frustrating experience, here are a few suggestions about moving Docker's storage.
(a) Be sure to stop the Docker service before making any changes.
(b) Use Docker's configuration file and the -g option to indicate the location of storage. This provides both flexibility and safety in trying different locations.
(c) If you use the cp command to copy storage contents, be sure to include the -p option to preserve owner, mode, and timestamp.
Here's an example of a sequence of steps that has worked in moving the Docker storage folder to a different location by using cp.
(1) Stop the Docker service.
# service docker stop
(2) Copy Docker's storage folder to a different partition.
# cd /var/lib
# cp -r -p docker /home/earl
# mv docker docker-save
(3) Edit the Docker configuration file to add the -g option to point to the new location. The file may already contain a commented line you can use as a starting point.
# vim /etc/default/docker
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /home/earl/docker"
(4) Re-start the Docker service. Run your container to test the change.
# service docker start
(5) When you are confident that the change is correct, you can remove the original directory to recover the space.
# rm -r /var/lib/docker-save
Sources:
How do I change the Docker image installation directory?
https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169
That sounded pretty simple, but it turned out to be a vexing problem.
After moving the directory, I ran a Docker container that contained an instance of Drupal 7. When I browsed to the Drupal site I got the error:
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'If you do a web search on this error message, there are a lot of possible causes. In my case, the Docker container had been running fine just before the directory move, so that was an obvious culprit.
I had typed in several cp commands before running the container again, but I didn't have a clear memory or record of the steps I had taken. After the first occurrence of the error, I did further copy commands. The error kept occurring, but once in a while the site came up okay. This thrashing about included my re-initializing the docker directory at least once.
From my frustrating experience, here are a few suggestions about moving Docker's storage.
(a) Be sure to stop the Docker service before making any changes.
(b) Use Docker's configuration file and the -g option to indicate the location of storage. This provides both flexibility and safety in trying different locations.
(c) If you use the cp command to copy storage contents, be sure to include the -p option to preserve owner, mode, and timestamp.
Here's an example of a sequence of steps that has worked in moving the Docker storage folder to a different location by using cp.
(1) Stop the Docker service.
# service docker stop
(2) Copy Docker's storage folder to a different partition.
# cd /var/lib
# cp -r -p docker /home/earl
# mv docker docker-save
(3) Edit the Docker configuration file to add the -g option to point to the new location. The file may already contain a commented line you can use as a starting point.
# vim /etc/default/docker
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /home/earl/docker"
(4) Re-start the Docker service. Run your container to test the change.
# service docker start
(5) When you are confident that the change is correct, you can remove the original directory to recover the space.
# rm -r /var/lib/docker-save
Sources:
How do I change the Docker image installation directory?
https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169
Aug 2, 2017
VirtualBox, Vagrant, and KVM
I was exploring the possibility of using Vagrant with VirtualBox for doing development on my local system.
The system is running Linux Mint 17, and I was using Vagrant 1.9.7 with VirtualBox 5.1.24.
I immediately starting having problems with some of the Vagrant boxes that I tried to run. (A "Vagrant box" is an initial machine image to be loaded into the virtual machine.)
For example, using the box hashicorp/precise64, the vagrant up command showed the error:
(1) The processor hardware does not have the capabilities required by VirtualBox to support Linux KVM (kernel-based virtual machines). These capabilities are either Intel's VT-x or AMD's AMD-V.
VT-x is sometimes encoded as vmx, and AMD-V is sometimes encoded as svm.
(2) The processor hardware has the capabilities but they are not enabled.
Without this hardware, VirtualBox cannot run 64-bit operating systems. However, it can still run 32-bit operating systems.
In my case, it turned out that I have a lower-end processor that does not have the capabilities at all. I was able to check this by looking at the file /proc/cpuinfo to find the model number of the processor:
model name : Intel(R) Pentium(R) CPU B960 @ 2.20GHz
Then using the model number B960 I searched this Intel site to find its specs.
If the processor has the capabilities but they are not enabled, then you might be able to enable them by going into the BIOS and looking for a setting such as VT (virtualization technology).
And for my next system, I will be looking for the processor to have this.
Sources:
ERROR: VT-X is not available
https://forums.virtualbox.org/viewtopic.php?f=8&t=17090
PRODUCT SPECIFICATIONS
https://ark.intel.com/#@Processors
KVM/Installation
https://help.ubuntu.com/community/KVM/Installation
x86 virtualization
https://en.wikipedia.org/wiki/X86_virtualization
The system is running Linux Mint 17, and I was using Vagrant 1.9.7 with VirtualBox 5.1.24.
I immediately starting having problems with some of the Vagrant boxes that I tried to run. (A "Vagrant box" is an initial machine image to be loaded into the virtual machine.)
For example, using the box hashicorp/precise64, the vagrant up command showed the error:
Stderr: VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)Later, after I had learned how to configure Vagrant to have VirtualBox display its own user window, I tried to run the box geerlingguy/ubuntu1604 and got this error from VirtualBox:
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
VT-x/AMD-V hardware acceleration is not available on your system. Your 64-bit guest will fail to detect a 64-bit CPU and will not be able to boot.And on the command line, kvm-ok showed:
INFO: Your CPU does not support KVM extensionsIt turns out that there are two possible explanations for these error messages.
KVM acceleration can NOT be used
(1) The processor hardware does not have the capabilities required by VirtualBox to support Linux KVM (kernel-based virtual machines). These capabilities are either Intel's VT-x or AMD's AMD-V.
VT-x is sometimes encoded as vmx, and AMD-V is sometimes encoded as svm.
(2) The processor hardware has the capabilities but they are not enabled.
Without this hardware, VirtualBox cannot run 64-bit operating systems. However, it can still run 32-bit operating systems.
In my case, it turned out that I have a lower-end processor that does not have the capabilities at all. I was able to check this by looking at the file /proc/cpuinfo to find the model number of the processor:
model name : Intel(R) Pentium(R) CPU B960 @ 2.20GHz
Then using the model number B960 I searched this Intel site to find its specs.
If the processor has the capabilities but they are not enabled, then you might be able to enable them by going into the BIOS and looking for a setting such as VT (virtualization technology).
And for my next system, I will be looking for the processor to have this.
Sources:
ERROR: VT-X is not available
https://forums.virtualbox.org/viewtopic.php?f=8&t=17090
PRODUCT SPECIFICATIONS
https://ark.intel.com/#@Processors
KVM/Installation
https://help.ubuntu.com/community/KVM/Installation
x86 virtualization
https://en.wikipedia.org/wiki/X86_virtualization
May 25, 2017
A Reason to Use the Drupal Coder / PHP_CodeSniffer utility
Drupal Coder includes the command line utility PHP_CodeSniffer, which parses source code to detect violations of a coding standard.
Actually, there are two utilities in the package, one to detect violations, a second to automatically make changes for those that can be so fixed.
I used the provided Drupal standard and ran it against all of the source code of the Optimizely module.
There are more than a hundred "sniffs" that are checked against. Individually, many are minor and relatively insignificant. A few are downright annoying. In the aggregate, though, I do feel that the resulting code was improved in terms of its readability.
The main benefit I've experienced so far is that I am being nudged into writing doc comments for all functions and classes. At first, I was resistent to doing so because good naming is often sufficient as documentation. As I edited file after file, though, I started to appreciate this kind of commenting as a desirable, consistent practice to adopt.
In one case, writing descriptions about a class and its methods helped me realize that the class was not entirely cohesive and maybe should have been written as two classes instead.
Retroactively applying the Drupal coding standard to the entire module was quite a bit of work. Moving forward, using PHP_CodeSniffer incrementally as a matter of habit should be much, much easier.
Sources:
Coder
https://www.drupal.org/project/coder
Installing Coder Sniffer
https://www.drupal.org/node/1419988
Actually, there are two utilities in the package, one to detect violations, a second to automatically make changes for those that can be so fixed.
I used the provided Drupal standard and ran it against all of the source code of the Optimizely module.
There are more than a hundred "sniffs" that are checked against. Individually, many are minor and relatively insignificant. A few are downright annoying. In the aggregate, though, I do feel that the resulting code was improved in terms of its readability.
The main benefit I've experienced so far is that I am being nudged into writing doc comments for all functions and classes. At first, I was resistent to doing so because good naming is often sufficient as documentation. As I edited file after file, though, I started to appreciate this kind of commenting as a desirable, consistent practice to adopt.
In one case, writing descriptions about a class and its methods helped me realize that the class was not entirely cohesive and maybe should have been written as two classes instead.
Retroactively applying the Drupal coding standard to the entire module was quite a bit of work. Moving forward, using PHP_CodeSniffer incrementally as a matter of habit should be much, much easier.
Sources:
Coder
https://www.drupal.org/project/coder
Installing Coder Sniffer
https://www.drupal.org/node/1419988
Apr 21, 2017
Using xDebug and Sublime Text with Docker
I've used xDebug with Sublime Text locally for quite some time but have started playing with Docker containers to instantiate instances of Apache, PHP, MySQL, and Drupal 8.
The Docker image I use has xDebug enabled for PHP, but I wanted to have xDebug running in the container to communicate with Sublime running on my local host system. This is not complicated, but it still took quite awhile for me to determine the correct settings.
Some of my confusion was due to the terms server and client as used in documentation and comments. Most of the time, server refers to xDebug running within PHP, and client refers to the IDE or text editor such as Sublime.
On the other hand, apparently it is xDebug that initiates the connection to the IDE, which makes xDebug act like a client. Also, xDebug has a setting called remote_host which sounds like a remote server that it is communicating with.
In the container I'm running, the xDebug settings are in
/etc/php5/mods-available/xdebug.ini
Here are tips to get these components working together.
• Run ifconfig in a local terminal to get the local ip address.
Working on my laptop connected to a home router, ifconfig shows eth0 with inet addr 10.0.0.3. I use that value as follows in xdebug.ini
xdebug.remote_host = 10.0.0.3
At a public library using their wi-fi, ifconfig shows wlan0 with inet addr 10.12.13.211, for example, but the address changes from session to session.
xdebug.remote_host = 10.12.13.211
This is the key difference from running in a purely local way without Docker, where localhost is a typically used value for the remote_host setting.
• Here are the settings that need to be present in xdebug.ini.
xdebug.remote_enable = On
xdebug.remote_host = 10.0.0.3
xdebug.remote_port = 9000
• If you have trouble getting your setup to work, use a log for xDebug to record errors and warnings. Enable the log by adding the following directive into your xdebug.ini file, e.g.
xdebug.remote_log = /tmp/xdebug.log
This is useful for debugging. For example, at the beginning of the log there will probably be an indication of whether xDebug is even able to connect to the client, which is an important clue.
But use this key only as needed since it can generate a lot of log messages, some of which seem spurious.
• For the Sublime editor, the key setting is path_mapping. Its value is an object that indicates corresponding paths. For example,
{ "/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/" }
The key (the left side) is a path in the Docker container where the xDebug server finds its source code. Its value (the right side) is the path in the local system where Sublime finds the corresponding code.
In my use case, I am only interested in the code for the Optimizely module, so I'm only providing a mapping between the two root directories of the module.
• Here are the settings that need to be present in Sublime for its xDebug package.
"port": 9000,
"path_mapping": { "/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/" },
If you need to do troubleshooting, you might add the following setting as well in order for Sublime to output messages to its own local xDebug log.
"debug": true
Also use this key only as needed since it can generate a lot of log messages, some of which seem spurious.
• No port forwarding is needed.
The Docker documentation states: "By default Docker containers can make connections to the outside world ..." Since it is xDebug within the container that initiates the connection to Sublime running outside, there is no need to use port forwarding for the port between them.
• Once you've got the correct settings for xdebug.ini, there are different ways to persist them. In my case, some settings don't change, but I work in different locations where the IP address for remote_host does vary, so I took a hybrid approach.
First, I used the docker commit command to capture the current state of a container into an image. In that container I had edited xdebug.ini with the settings that remain the same.
# docker commit distracted_dijkstra drupal-xdebug
Second, I use the docker run command with the -e option to provide the IP address as an environment variable when instantiating the image in a new container.
# docker run -e XDEBUG_CONFIG="remote_host=10.0.0.3" drupal-xdebug
Sources:
Xdebug 2 | Remote Debugging
https://xdebug.org/docs/remote#browser_session
martomo / SublimeTextXdebug
https://github.com/martomo/SublimeTextXdebug/blob/master/Xdebug.sublime-settings
Debug your PHP in Docker with Intellij/PHPStorm and Xdebug
https://gist.github.com/chadrien/c90927ec2d160ffea9c4
The Docker image I use has xDebug enabled for PHP, but I wanted to have xDebug running in the container to communicate with Sublime running on my local host system. This is not complicated, but it still took quite awhile for me to determine the correct settings.
Some of my confusion was due to the terms server and client as used in documentation and comments. Most of the time, server refers to xDebug running within PHP, and client refers to the IDE or text editor such as Sublime.
On the other hand, apparently it is xDebug that initiates the connection to the IDE, which makes xDebug act like a client. Also, xDebug has a setting called remote_host which sounds like a remote server that it is communicating with.
In the container I'm running, the xDebug settings are in
/etc/php5/mods-available/xdebug.ini
Here are tips to get these components working together.
• Run ifconfig in a local terminal to get the local ip address.
Working on my laptop connected to a home router, ifconfig shows eth0 with inet addr 10.0.0.3. I use that value as follows in xdebug.ini
xdebug.remote_host = 10.0.0.3
At a public library using their wi-fi, ifconfig shows wlan0 with inet addr 10.12.13.211, for example, but the address changes from session to session.
xdebug.remote_host = 10.12.13.211
This is the key difference from running in a purely local way without Docker, where localhost is a typically used value for the remote_host setting.
• Here are the settings that need to be present in xdebug.ini.
xdebug.remote_enable = On
xdebug.remote_host = 10.0.0.3
xdebug.remote_port = 9000
• If you have trouble getting your setup to work, use a log for xDebug to record errors and warnings. Enable the log by adding the following directive into your xdebug.ini file, e.g.
xdebug.remote_log = /tmp/xdebug.log
This is useful for debugging. For example, at the beginning of the log there will probably be an indication of whether xDebug is even able to connect to the client, which is an important clue.
But use this key only as needed since it can generate a lot of log messages, some of which seem spurious.
• For the Sublime editor, the key setting is path_mapping. Its value is an object that indicates corresponding paths. For example,
{ "/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/" }
The key (the left side) is a path in the Docker container where the xDebug server finds its source code. Its value (the right side) is the path in the local system where Sublime finds the corresponding code.
In my use case, I am only interested in the code for the Optimizely module, so I'm only providing a mapping between the two root directories of the module.
• Here are the settings that need to be present in Sublime for its xDebug package.
"port": 9000,
"path_mapping": { "/var/www/modules/custom/optimizely/": "/var/www/html/opti/modules/contrib/optimizely/" },
If you need to do troubleshooting, you might add the following setting as well in order for Sublime to output messages to its own local xDebug log.
"debug": true
Also use this key only as needed since it can generate a lot of log messages, some of which seem spurious.
• No port forwarding is needed.
The Docker documentation states: "By default Docker containers can make connections to the outside world ..." Since it is xDebug within the container that initiates the connection to Sublime running outside, there is no need to use port forwarding for the port between them.
• Once you've got the correct settings for xdebug.ini, there are different ways to persist them. In my case, some settings don't change, but I work in different locations where the IP address for remote_host does vary, so I took a hybrid approach.
First, I used the docker commit command to capture the current state of a container into an image. In that container I had edited xdebug.ini with the settings that remain the same.
# docker commit distracted_dijkstra drupal-xdebug
Second, I use the docker run command with the -e option to provide the IP address as an environment variable when instantiating the image in a new container.
# docker run -e XDEBUG_CONFIG="remote_host=10.0.0.3" drupal-xdebug
Sources:
Xdebug 2 | Remote Debugging
https://xdebug.org/docs/remote#browser_session
martomo / SublimeTextXdebug
https://github.com/martomo/SublimeTextXdebug/blob/master/Xdebug.sublime-settings
Debug your PHP in Docker with Intellij/PHPStorm and Xdebug
https://gist.github.com/chadrien/c90927ec2d160ffea9c4
Apr 17, 2017
Initial Thoughts on Using Docker
I have started to use the Docker images from wadmiraal/drupal for local Drupal development, for example, wadmiraal/drupal:8.1.0 to use Drupal 8.1.0. These images are very well documented at Use Docker to kickstart your Drupal development.
There are other images for working with Drupal, but I happened to choose this set since it incorporates the versions of PHP, MySQL, and Apache that are close to what I've been using.
Here are some random notes on what I have experienced initially as someone who is new to Docker.
(1) Containers are easy and really fast to spin up
(At least on my Ubuntu-based system).
If you want to start completely fresh, run a new container, which means any changes you have made are lost. Sometimes, that's what you want, for example, when I want a fresh install of the module I'm working on and a clean database.
On the other hand, if you want to preserve changes to the file system of the container, you can stop it and then start the same container later. But be aware that it's really easy to accumulate clutter in the way of containers that you no longer want and have to manually remove. You can see all containers, running or not, by the command: docker ps -a
(2) There are different ways to communicate into Docker containers.
The ways I've used are port forwarding and volume mounting.
With port forwarding, when you run a container you specify which ports on the local host are passed on to a corresponding port of the container. For example, port 8080 locally can be mapped to the default port 80 of the container for http. Browsing to an address such as localhost:8080 then sends the request to the instance of the web server running in the container.
Volume mounting is a way to make local directories visible to processes running in the container. I mount my local development directory for the Optimizely module to a directory path in the container's file system under the Drupal site. This allows me to edit code locally without having to do so inside the container. Nice!
(3) Using ssh and scp
If you run the container with the appropriate port forwarding, you can ssh into the container. In the case of the image wadmiraal/drupal, once you have an ssh terminal the vi and vim editors are available.
However, other tools that you might want are not there. These can be added on the fly by using apt-get install, for example. But keep in mind that such changes will not necessarily persist, depending on how you manage the container.
If ssh is working, then so does scp for copying files back and forth between host and container. I have a one line php script that calls the function phpinfo(), which I scp from my local into the web root of the container for troubleshoot.
(4) Using xDebug and Sublime Text with Docker
The Docker image has xDebug enabled for PHP, but I'd like to have xDebug running on the container to communicate with Sublime on the local system so that I can edit and step through code locally.
So far, I am struggling to set this up. I expect to crack this nut eventually and will blog about it when I do.
Sources:
Docker Overview
https://docs.docker.com/engine/understanding-docker/
Docker Tutorial Series, Part 1: An Introduction | Docker Components
http://blog.flux7.com/blogs/docker/docker-tutorial-series-part-1-an-introduction
Use Docker to kickstart your Drupal development
http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/
Bind container ports to the host
https://docs.docker.com/engine/userguide/networking/default_network/binding/
There are other images for working with Drupal, but I happened to choose this set since it incorporates the versions of PHP, MySQL, and Apache that are close to what I've been using.
Here are some random notes on what I have experienced initially as someone who is new to Docker.
(1) Containers are easy and really fast to spin up
(At least on my Ubuntu-based system).
If you want to start completely fresh, run a new container, which means any changes you have made are lost. Sometimes, that's what you want, for example, when I want a fresh install of the module I'm working on and a clean database.
On the other hand, if you want to preserve changes to the file system of the container, you can stop it and then start the same container later. But be aware that it's really easy to accumulate clutter in the way of containers that you no longer want and have to manually remove. You can see all containers, running or not, by the command: docker ps -a
(2) There are different ways to communicate into Docker containers.
The ways I've used are port forwarding and volume mounting.
With port forwarding, when you run a container you specify which ports on the local host are passed on to a corresponding port of the container. For example, port 8080 locally can be mapped to the default port 80 of the container for http. Browsing to an address such as localhost:8080 then sends the request to the instance of the web server running in the container.
Volume mounting is a way to make local directories visible to processes running in the container. I mount my local development directory for the Optimizely module to a directory path in the container's file system under the Drupal site. This allows me to edit code locally without having to do so inside the container. Nice!
(3) Using ssh and scp
If you run the container with the appropriate port forwarding, you can ssh into the container. In the case of the image wadmiraal/drupal, once you have an ssh terminal the vi and vim editors are available.
However, other tools that you might want are not there. These can be added on the fly by using apt-get install, for example. But keep in mind that such changes will not necessarily persist, depending on how you manage the container.
If ssh is working, then so does scp for copying files back and forth between host and container. I have a one line php script that calls the function phpinfo(), which I scp from my local into the web root of the container for troubleshoot.
(4) Using xDebug and Sublime Text with Docker
The Docker image has xDebug enabled for PHP, but I'd like to have xDebug running on the container to communicate with Sublime on the local system so that I can edit and step through code locally.
So far, I am struggling to set this up. I expect to crack this nut eventually and will blog about it when I do.
Sources:
Docker Overview
https://docs.docker.com/engine/understanding-docker/
Docker Tutorial Series, Part 1: An Introduction | Docker Components
http://blog.flux7.com/blogs/docker/docker-tutorial-series-part-1-an-introduction
Use Docker to kickstart your Drupal development
http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/
Bind container ports to the host
https://docs.docker.com/engine/userguide/networking/default_network/binding/
Nov 22, 2016
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes)
I ran into the above error message while modifying some custom Drupal code. At first, I thought I just needed to increase the memory limit due to some newly called core functions consuming more than what was allocated.
There's good discussion about different ways to change the PHP memory limit in the article Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...
On my local stack, one relevant file is /etc/php5/apache2/php.ini which had a limit of 512M that I had set about two years ago. That seemed like a lot already. I upped it to 640M, but got the same error at exactly the same line of code.
So I fired up a debugger to trace through. When execution reached the loop body containing the reported line where execution died (line 4 below), I single-stepped, single-stepped, single-stepped, ... and it didn't leave after the expected number of iterations.
∞
Sources:
Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...
https://www.drupal.org/node/76156
There's good discussion about different ways to change the PHP memory limit in the article Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...
On my local stack, one relevant file is /etc/php5/apache2/php.ini which had a limit of 512M that I had set about two years ago. That seemed like a lot already. I upped it to 640M, but got the same error at exactly the same line of code.
So I fired up a debugger to trace through. When execution reached the loop body containing the reported line where execution died (line 4 below), I single-stepped, single-stepped, single-stepped, ... and it didn't leave after the expected number of iterations.
1: $dirname = pathinfo($current_path, PATHINFO_DIRNAME);Line 3 kept adding entries to the $page array until memory was exhausted. It was my own coding mistake in inadvertently creating a runaway infinite loop.
2: while ($dirname && $dirname != '.') {
3: $page['#cache']['tags'][] = 'optimizely:' . $dirname . '/*';
4: $dirname = pathinfo($dirname, PATHINFO_DIRNAME);
5: }
∞
Sources:
Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...
https://www.drupal.org/node/76156
Nov 3, 2016
Using Drupal 8 Cache Tags for Page Caching
This is a small case study in converting how page caching is done in the D7 version of the Optimizely module to Drupal 8.
The purpose of the module is to manage the insertion of certain <script> elements into designated pages of a site. To do so, the user creates one or more projects. Each project has one or more url paths. When a project is enabled, all pages that match one of its paths have the <script> element added.
To specify project paths, use of a trailing * wildcard is allowed, as are special page designators. For example, these are all valid paths.
/node/2
/node/*
/admin/config/system
/admin/*
*
<front>
In the case of /admin/* it would match against any of
/admin/
/admin/config/
/admin/config/system/
/admin/people
/admin/people/create
...
In the D7 version, invalidating is done through calls to cache_clear_all(), which takes three parameters. The function is used by the module in three different ways.
(1) To invalidate a particular page, e.g.
cache_clear_all('/node/2', 'cache_page', FALSE);
(2) To invalidate a path with a trailing wildcard, e.g.
cache_clear_all('/node/*', 'cache_page', TRUE);
(3) To invalidate all pages of the site,
cache_clear_all('*', 'cache_page', TRUE);
In Drupal 8 the Cache API is completely different. Function cache_clear_all() has disappeared.
After some research, it looked like using cache tags would be the way to go. The article Cacheability of render arrays was especially helpful in how to think about caching as applied to page rendering.
There are two facets to implementing this. The first is what needs to be done when a page is rendered, the second is what to invalidate when triggering changes occur.
For page rendering, I already had an implementation of hook_page_attachments() that checked for inserting the element into any page whose path matched against any of the enabled project paths.
This hook function is where cache tags could be added to the page. This turned out to be a little tricky. A page must be invalidated for two different use cases: when the page contains the element which now needs to be removed, and when the page does not contain the element but it now needs to be added.
For the first case, I decided to just use the matching project path act as the cache tag (there can only be one because overlapping project paths are not allowed).
But for the second case, I had to cover all the possible project paths that might be enabled in the future. So, for example, for the page at /node/2 there are three such possible paths.
/node/2
/node/*
*
For every page rendered, it was necessary to attach all of these possible project paths as cache tags. Here is a snippet of code that shows how this is done in hook_page_attachments().
// Site-wide wildcard.
$page['#cache']['tags'][] = 'optimizely:*';
// Non site-wide wildcards. Repeat for every directory level.
$dirname = pathinfo($current_path, PATHINFO_DIRNAME);
while ($dirname && $dirname != '/') {
$page['#cache']['tags'][] = 'optimizely:' . $dirname . '/*';
$dirname = pathinfo($dirname, PATHINFO_DIRNAME);
}
// The specific page url.
$page['#cache']['tags'][] = 'optimizely:' . $current_path;
// Finally, if there is an alias for the page, tag it.
if ($current_path_alias) {
$page['#cache']['tags'][] = 'optimizely:' . $current_path_alias;
}
The optimizely: prefix follows the convention of prefixing a group name as part of the tag where appropriate. For example, cache tags from core include node:1 and config:node.type.article.
Finally, there is the matter of what and how to invalidate when changes to the projects and their paths are submitted. This turned out to be fairly easy to implement.
An array of all relevant project paths is passed to a function that carries out the following.
$cache_tags = [];
foreach ($path_array as $path) {
$cache_tags[] = 'optimizely:' . $path;
}
\Drupal::service('cache_tags.invalidator')->invalidateTags($cache_tags);
For debugging purposes, outputting X-Drupal-Cache-Tags in HTTP headers was extremely useful. See my earlier post Enable and Use X-Drupal-Cache-Tags in HTTP headers.
This post is about the caching that is done by Drupal itself. The D7 version also checks for the presence of the varnish module and calls a function of that module if it exists. I did not pursue a replacement for that functionality, but the articles Varnish and Use Drupal 8 Cache Tags with Varnish and Purge look promising.
Sources:
Function cache_clear_all() has been removed
https://optimizely-to-drupal-8.blogspot.com/2014/07/function-cacheclearall-has-been-removed.html
Cacheability of render arrays
https://www.drupal.org/developing/api/8/render/arrays/cacheability
Cache tags
https://www.drupal.org/developing/api/8/cache/tags
Allow to set #cache metadata in hook_page_attachments() https://www.drupal.org/node/2475749
public static function Cache::invalidateTags
https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Cache!Cache.php/function/Cache%3A%3AinvalidateTags/8
The purpose of the module is to manage the insertion of certain <script> elements into designated pages of a site. To do so, the user creates one or more projects. Each project has one or more url paths. When a project is enabled, all pages that match one of its paths have the <script> element added.
To specify project paths, use of a trailing * wildcard is allowed, as are special page designators. For example, these are all valid paths.
/node/2
/node/*
/admin/config/system
/admin/*
*
<front>
In the case of /admin/* it would match against any of
/admin/
/admin/config/
/admin/config/system/
/admin/people
/admin/people/create
...
In the D7 version, invalidating is done through calls to cache_clear_all(), which takes three parameters. The function is used by the module in three different ways.
(1) To invalidate a particular page, e.g.
cache_clear_all('/node/2', 'cache_page', FALSE);
(2) To invalidate a path with a trailing wildcard, e.g.
cache_clear_all('/node/*', 'cache_page', TRUE);
(3) To invalidate all pages of the site,
cache_clear_all('*', 'cache_page', TRUE);
In Drupal 8 the Cache API is completely different. Function cache_clear_all() has disappeared.
After some research, it looked like using cache tags would be the way to go. The article Cacheability of render arrays was especially helpful in how to think about caching as applied to page rendering.
There are two facets to implementing this. The first is what needs to be done when a page is rendered, the second is what to invalidate when triggering changes occur.
For page rendering, I already had an implementation of hook_page_attachments() that checked for inserting the element into any page whose path matched against any of the enabled project paths.
This hook function is where cache tags could be added to the page. This turned out to be a little tricky. A page must be invalidated for two different use cases: when the page contains the element which now needs to be removed, and when the page does not contain the element but it now needs to be added.
For the first case, I decided to just use the matching project path act as the cache tag (there can only be one because overlapping project paths are not allowed).
But for the second case, I had to cover all the possible project paths that might be enabled in the future. So, for example, for the page at /node/2 there are three such possible paths.
/node/2
/node/*
*
For every page rendered, it was necessary to attach all of these possible project paths as cache tags. Here is a snippet of code that shows how this is done in hook_page_attachments().
// Site-wide wildcard.
$page['#cache']['tags'][] = 'optimizely:*';
// Non site-wide wildcards. Repeat for every directory level.
$dirname = pathinfo($current_path, PATHINFO_DIRNAME);
while ($dirname && $dirname != '/') {
$page['#cache']['tags'][] = 'optimizely:' . $dirname . '/*';
$dirname = pathinfo($dirname, PATHINFO_DIRNAME);
}
// The specific page url.
$page['#cache']['tags'][] = 'optimizely:' . $current_path;
// Finally, if there is an alias for the page, tag it.
if ($current_path_alias) {
$page['#cache']['tags'][] = 'optimizely:' . $current_path_alias;
}
The optimizely: prefix follows the convention of prefixing a group name as part of the tag where appropriate. For example, cache tags from core include node:1 and config:node.type.article.
Finally, there is the matter of what and how to invalidate when changes to the projects and their paths are submitted. This turned out to be fairly easy to implement.
An array of all relevant project paths is passed to a function that carries out the following.
$cache_tags = [];
foreach ($path_array as $path) {
$cache_tags[] = 'optimizely:' . $path;
}
\Drupal::service('cache_tags.invalidator')->invalidateTags($cache_tags);
For debugging purposes, outputting X-Drupal-Cache-Tags in HTTP headers was extremely useful. See my earlier post Enable and Use X-Drupal-Cache-Tags in HTTP headers.
This post is about the caching that is done by Drupal itself. The D7 version also checks for the presence of the varnish module and calls a function of that module if it exists. I did not pursue a replacement for that functionality, but the articles Varnish and Use Drupal 8 Cache Tags with Varnish and Purge look promising.
Sources:
Function cache_clear_all() has been removed
https://optimizely-to-drupal-8.blogspot.com/2014/07/function-cacheclearall-has-been-removed.html
Cacheability of render arrays
https://www.drupal.org/developing/api/8/render/arrays/cacheability
Cache tags
https://www.drupal.org/developing/api/8/cache/tags
Allow to set #cache metadata in hook_page_attachments() https://www.drupal.org/node/2475749
public static function Cache::invalidateTags
https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Cache!Cache.php/function/Cache%3A%3AinvalidateTags/8
Oct 17, 2016
Enable and Use X-Drupal-Cache-Tags in HTTP headers
While converting from Drupal 7's caching to that of Drupal 8, for debugging purposes I wanted to enable the display of X-Drupal-Cache-Tags in HTTP headers.
How to do this is documented in the article CacheableResponseInterface and is actually simple, but I had enough trouble with getting this to work that I'm providing a few of my own notes here.
In Drupal 8 core, there is the file /sites/default/default.services.yml Copy this file to the same directory, creating a new file named services.yml (assuming you don't already have such a file).
For our purposes, in services.yml the only key you need is http.response.debug_cacheability_headers.
In my situation, I ended up with a services.yml that only contained the following two lines, where the value of the key is changed to true. I deleted all of the other keys.
parameters:
http.response.debug_cacheability_headers: true
After creating and/or editing services.yml be sure to do "a container rebuild, which is necessary when changing a container parameter". One way to accomplish this is via Clear all caches in the admin UI.
Finally, to see the display of X-Drupal-Cache-Tags in HTTP headers, I use the Chrome browser on a Linux system. It happens to be version 39.0.
Navigate to: More tools > Developer tools > Network tab
After loading a page whose cache tags you are interested in, on the left side of the Developer tools panel, click on the url for the page. Then on the right, click on the Headers subtab and look under Response Headers for the X-Drupal-Cache-Tags property. You will see a number of cache tags that come with Drupal core in addition to your own custom ones.
Source:
CacheableResponseInterface
https://www.drupal.org/developing/api/8/response/cacheable-response-interface#debugging
How to do this is documented in the article CacheableResponseInterface and is actually simple, but I had enough trouble with getting this to work that I'm providing a few of my own notes here.
In Drupal 8 core, there is the file /sites/default/default.services.yml Copy this file to the same directory, creating a new file named services.yml (assuming you don't already have such a file).
For our purposes, in services.yml the only key you need is http.response.debug_cacheability_headers.
In my situation, I ended up with a services.yml that only contained the following two lines, where the value of the key is changed to true. I deleted all of the other keys.
parameters:
http.response.debug_cacheability_headers: true
After creating and/or editing services.yml be sure to do "a container rebuild, which is necessary when changing a container parameter". One way to accomplish this is via Clear all caches in the admin UI.
Finally, to see the display of X-Drupal-Cache-Tags in HTTP headers, I use the Chrome browser on a Linux system. It happens to be version 39.0.
Navigate to: More tools > Developer tools > Network tab
After loading a page whose cache tags you are interested in, on the left side of the Developer tools panel, click on the url for the page. Then on the right, click on the Headers subtab and look under Response Headers for the X-Drupal-Cache-Tags property. You will see a number of cache tags that come with Drupal core in addition to your own custom ones.
Source:
CacheableResponseInterface
https://www.drupal.org/developing/api/8/response/cacheable-response-interface#debugging
Aug 30, 2016
Call of MySQL database function fails
This post describes (another) database issue that was tracked down by my colleague Luis Delacruz.
We had both a live site and a corresponding development site using the same MySQL database server. On the live site, carrying out a particular user task worked fine, but doing the same task on the development site would fail with the message "DB Error: unknown error".
The problem was eventually traced back to the fact that the contents of the database for the development site had been loaded by importing a backup of the live site.
The live site contained several database functions. For example, here's one of the exported function definitions.
CREATE DEFINER=`user1`@`localhost`
FUNCTION `clean_string`(in_str varchar(4096))
RETURNS varchar(4096) CHARSET latin1
BEGIN
/**
* Function will strip all non-ASCII and unwanted ASCII characters in string
*
* @author Shay Anderson 10.11
*
* @param VARCHAR in_arg
* @return VARCHAR
*/
DECLARE i, len SMALLINT DEFAULT 1;
DECLARE ret CHAR(255) DEFAULT '';
DECLARE c CHAR(1);
SET len = CHAR_LENGTH( in_str );
REPEAT
BEGIN
SET c = MID( in_str, i, 1 );
IF c REGEXP '[[:alnum:]]' THEN
SET ret=CONCAT(ret,c);
END IF;
SET i = i + 1;
END;
UNTIL i > len END REPEAT;
RETURN ret;
END
Luis noticed that the definitions of this database function as well as others included the clause DEFINER = 'user1'@'localhost'.
By default, the SQL SECURITY characteristic of a function is DEFINER. That means that when the routine is executed, it does so within the security context of the user specified as the DEFINER.
This worked fine for the live site because its database runs under user user1.
However, for the staging site, that user does not have access to the staging database, so it was denied. That is, the user did not have sufficient permissions for executing the function body as applied to the other database.
Deleting the DEFINER clause and reloading the function confirmed that the clause was the problem.
Also, a key factor was that both databases are managed by the same db server, within which user1 exists. If the function definition were imported into a different db server for which user1 does not exist, it could be that the problem would not occur.
Further research uncovered this advice:
"For a stored routine or view, use SQL SECURITY INVOKER in the object definition when possible so that it can be used only by users with permissions appropriate for the operations performed by the object. ".
In other words, set the SQL SECURITY characteristic to INVOKER so that the security context of whichever user invokes the function is in effect. This seems ideal.
So we modified the beginning part of the function definition as follows:
CREATEDEFINER=`user1`@`localhost`
FUNCTION `clean_string`(in_str varchar(4096))
RETURNS varchar(4096) CHARSET latin1
SQL SECURITY INVOKER
BEGIN
....
END
(Incidentally, the error message "DB Error: unknown error" is apparently output by the Drupal 7 CiviCRM module.)
Sources:
Access Control for Stored Programs and Views
https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html
We had both a live site and a corresponding development site using the same MySQL database server. On the live site, carrying out a particular user task worked fine, but doing the same task on the development site would fail with the message "DB Error: unknown error".
The problem was eventually traced back to the fact that the contents of the database for the development site had been loaded by importing a backup of the live site.
The live site contained several database functions. For example, here's one of the exported function definitions.
CREATE DEFINER=`user1`@`localhost`
FUNCTION `clean_string`(in_str varchar(4096))
RETURNS varchar(4096) CHARSET latin1
BEGIN
/**
* Function will strip all non-ASCII and unwanted ASCII characters in string
*
* @author Shay Anderson 10.11
*
* @param VARCHAR in_arg
* @return VARCHAR
*/
DECLARE i, len SMALLINT DEFAULT 1;
DECLARE ret CHAR(255) DEFAULT '';
DECLARE c CHAR(1);
SET len = CHAR_LENGTH( in_str );
REPEAT
BEGIN
SET c = MID( in_str, i, 1 );
IF c REGEXP '[[:alnum:]]' THEN
SET ret=CONCAT(ret,c);
END IF;
SET i = i + 1;
END;
UNTIL i > len END REPEAT;
RETURN ret;
END
Luis noticed that the definitions of this database function as well as others included the clause DEFINER = 'user1'@'localhost'.
By default, the SQL SECURITY characteristic of a function is DEFINER. That means that when the routine is executed, it does so within the security context of the user specified as the DEFINER.
This worked fine for the live site because its database runs under user user1.
However, for the staging site, that user does not have access to the staging database, so it was denied. That is, the user did not have sufficient permissions for executing the function body as applied to the other database.
Deleting the DEFINER clause and reloading the function confirmed that the clause was the problem.
Also, a key factor was that both databases are managed by the same db server, within which user1 exists. If the function definition were imported into a different db server for which user1 does not exist, it could be that the problem would not occur.
Further research uncovered this advice:
"For a stored routine or view, use SQL SECURITY INVOKER in the object definition when possible so that it can be used only by users with permissions appropriate for the operations performed by the object. ".
In other words, set the SQL SECURITY characteristic to INVOKER so that the security context of whichever user invokes the function is in effect. This seems ideal.
So we modified the beginning part of the function definition as follows:
CREATE
FUNCTION `clean_string`(in_str varchar(4096))
RETURNS varchar(4096) CHARSET latin1
SQL SECURITY INVOKER
BEGIN
....
END
(Incidentally, the error message "DB Error: unknown error" is apparently output by the Drupal 7 CiviCRM module.)
Sources:
Access Control for Stored Programs and Views
https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html
May 28, 2016
Two sites on the same database server - PDOException: SQLSTATE[HY000][1129] Host 'nnn.nnn.nnn.nnn' is blocked because of many connection errors
A web search shows a number of forums and postings about the error message in the title of this post, but our particular problem involved two sites on the same web server and the same database server. One of the sites appeared to repeatedly bring down the other site.
We had had a working live site implemented in Drupal and wanted to add a separate staging site in order to have a more transparent workflow that is less risky.
So for the staging site, a separate Drupal instance was installed and a separate Drupal database was created using the same respective servers as for the live site.
The sites use the CiviCRM module, which has its own civicrm.settings.php file for configuration.
- - - - -
After staging was created, testing of the staging site was showing very odd errors in the form of inconsistent user profiles and such.
With help from the primary developers, I was able to track down that the civicrm.settings.php file was incorrectly specifying the live database for use by the staging site. Instead, it should have been specifying the staging database.
I corrected the configuration in civicrm.settings.php so that staging was accessing its own database via its own database user.
Here are the original constants defined in civicrm.settings.php
// These ip addresses are not the actual ones.
define( 'CIVICRM_UF_DSN', 'mysql://live_db_user:user_password@12.210.138.193/live_db?new_link=true' );
define( 'CIVICRM_DSN', 'mysql://live_db_user:user_password@12.210.138.193/live_db?new_link=true' );
And the revised constants in civicrm.settings.php
// These ip addresses are not the actual ones.
define( 'CIVICRM_UF_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' );
define( 'CIVICRM_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' );
I then emailed our tester about the changes late at night, without doing any of my own testing. Mea culpa.
That's when things got interesting.
- - - - -
The next morning, both the staging and the live sites were broken with the same error even though I had not changed any code on live.
PDOException: SQLSTATE[HY000][1129] Host '12.210.138.193' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' in lock_may_be_available() (line 167 of /var/www/oursite.org/httpdocs/includes/lock.inc).
Since the error message stated "unblock with 'mysqladmin flush-hosts' ", I logged onto the database server and did flush-hosts on the live database in order to keep the live site running. Live came back up. And I sighed with relief thinking that was enough.
Then I browsed to the staging site again to troubleshoot it. Now it came up with a different error.
PDOException: SQLSTATE[28000] [1045] Access denied for user 'staging_db_user'@'12.210.138.193' (using password: YES) in lock_may_be_available() (line 167 of /var/www/staging.oursite.org/httpdocs/includes/lock.inc).
A bit later, I browsed to the live site again. And it was broken again.
The pattern was: (1) fix the live site by running the flush-hosts command, (2) browse to the staging site, (3) and the live site is broken again.
Since keeping the live site running was a priority, this was driving me crazy!
Those of you who are more familiar with MySQL than I was may already know what had happened. With hindsight, the issue is so obvious that it makes me reflect on my own thinking processes.
- - - - -
It was my colleague Luis who cracked this. The following explanation is due to him.
The basic problem is that the database user staging_db_user did not have privileges to access the database via the 12.210.138.193 host address (although it did have access through a different ip). So whenever I browsed to the staging site, connection errors would occur.
The number of connection errors would quickly accumulate and exceed the allowed maximum. The relevant variable in MySQL is max_connect_errors, with a default value of something like 1000.
When the maximum number of errors was reached, the MySQL server would then block any further connection requests from that host ip, including for the live site!
This explains why browsing to the staging site would result in access errors for the live site as well. Because of the common ip, the two sites were inadvertently coupled and mutually dependent.
In the end, we decided to use a different host ip address for the staging site to access the database precisely to avoid this cross-site brittleness.
Source:
Troubleshooting Problems Connecting to MySQL
https://dev.mysql.com/doc/refman/5.7/en/problems-connecting.html
We had had a working live site implemented in Drupal and wanted to add a separate staging site in order to have a more transparent workflow that is less risky.
So for the staging site, a separate Drupal instance was installed and a separate Drupal database was created using the same respective servers as for the live site.
The sites use the CiviCRM module, which has its own civicrm.settings.php file for configuration.
- - - - -
After staging was created, testing of the staging site was showing very odd errors in the form of inconsistent user profiles and such.
With help from the primary developers, I was able to track down that the civicrm.settings.php file was incorrectly specifying the live database for use by the staging site. Instead, it should have been specifying the staging database.
I corrected the configuration in civicrm.settings.php so that staging was accessing its own database via its own database user.
Here are the original constants defined in civicrm.settings.php
// These ip addresses are not the actual ones.
define( 'CIVICRM_UF_DSN', 'mysql://live_db_user:user_password@12.210.138.193/live_db?new_link=true' );
define( 'CIVICRM_DSN', 'mysql://live_db_user:user_password@12.210.138.193/live_db?new_link=true' );
And the revised constants in civicrm.settings.php
// These ip addresses are not the actual ones.
define( 'CIVICRM_UF_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' );
define( 'CIVICRM_DSN', 'mysql://staging_db_user:user_password@12.210.138.193/staging_db?new_link=true' );
I then emailed our tester about the changes late at night, without doing any of my own testing. Mea culpa.
That's when things got interesting.
- - - - -
The next morning, both the staging and the live sites were broken with the same error even though I had not changed any code on live.
PDOException: SQLSTATE[HY000][1129] Host '12.210.138.193' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' in lock_may_be_available() (line 167 of /var/www/oursite.org/httpdocs/includes/lock.inc).
Since the error message stated "unblock with 'mysqladmin flush-hosts' ", I logged onto the database server and did flush-hosts on the live database in order to keep the live site running. Live came back up. And I sighed with relief thinking that was enough.
Then I browsed to the staging site again to troubleshoot it. Now it came up with a different error.
PDOException: SQLSTATE[28000] [1045] Access denied for user 'staging_db_user'@'12.210.138.193' (using password: YES) in lock_may_be_available() (line 167 of /var/www/staging.oursite.org/httpdocs/includes/lock.inc).
A bit later, I browsed to the live site again. And it was broken again.
The pattern was: (1) fix the live site by running the flush-hosts command, (2) browse to the staging site, (3) and the live site is broken again.
Since keeping the live site running was a priority, this was driving me crazy!
Those of you who are more familiar with MySQL than I was may already know what had happened. With hindsight, the issue is so obvious that it makes me reflect on my own thinking processes.
- - - - -
It was my colleague Luis who cracked this. The following explanation is due to him.
The basic problem is that the database user staging_db_user did not have privileges to access the database via the 12.210.138.193 host address (although it did have access through a different ip). So whenever I browsed to the staging site, connection errors would occur.
The number of connection errors would quickly accumulate and exceed the allowed maximum. The relevant variable in MySQL is max_connect_errors, with a default value of something like 1000.
When the maximum number of errors was reached, the MySQL server would then block any further connection requests from that host ip, including for the live site!
This explains why browsing to the staging site would result in access errors for the live site as well. Because of the common ip, the two sites were inadvertently coupled and mutually dependent.
In the end, we decided to use a different host ip address for the staging site to access the database precisely to avoid this cross-site brittleness.
Source:
Troubleshooting Problems Connecting to MySQL
https://dev.mysql.com/doc/refman/5.7/en/problems-connecting.html
Mar 15, 2016
Blocking Chinese and Korean spam for user-submitted content
I help maintain a Drupal 7 site called Power Poetry that is a platform for publishing poetry. It's really a great site with awesome writing that has grown steadily in both the amount of content and the number of page views.
Unfortunately, its popularity has apparently attracted some foreign spam content, particularly in Chinese and Korean.
Our initial take was that being U.S.-centric, we should limit submissions to only English and Spanish. So I did some research into language detection, found a service with an API that looked promising, and did an initial implementation. However, for technical reasons that I never did uncover, that API was not working on our hosting setup (even though it worked on my local system!).
We then changed our focus from language detection to the fact that our current problem was almost entirely due to submissions made in certain scripts (specific human writing systems).
I found that regular expressions in PHP support Unicode character properties. Those property codes include values that designate the Chinese script (which includes both traditional and simplified characters) and the Korean script.
We decided to take a zero-tolerance approach. A single Chinese or Korean character causes the user submission not to validate. So far, implementing this has drastically reduced the amount of inappropriate content and comments.
Here's the code, which is called by our Drupal form validation functions.
/**
* Check whether a string contains any characters from a
* banned script, such as Chinese or Korean.
*
* @param string $text
* The piece of text to be checked.
*
* @return TRUE | FALSE
* Returns TRUE if any Chinese or Korean characters detected.
* Otherwise, returns FALSE.
*/
function _contains_banned_scripts($text) {
$unicode_modifier = 'u';
// Detect whether there are any Chinese characters.
$chinese_regex = '\p{Han}+';
$preg_regex = '/' . $chinese_regex . '/' . $unicode_modifier;
$chinese_found = preg_match($preg_regex, $text);
if ($chinese_found == 1) {
return TRUE;
}
// Detect whether there are any Korean characters.
$korean_regex = '\p{Hangul}+';
$preg_regex = '/' . $korean_regex . '/' . $unicode_modifier;
$korean_found = preg_match($preg_regex, $text);
if ($korean_found == 1) {
return TRUE;
}
return FALSE;
}
Sources:
Forum spam
https://en.wikipedia.org/wiki/Forum_spam
Unicode Regular Expressions
http://www.regular-expressions.info/unicode.html
Unicode Character Properties
https://secure.php.net/manual/en/regexp.reference.unicode.php
Unfortunately, its popularity has apparently attracted some foreign spam content, particularly in Chinese and Korean.
Our initial take was that being U.S.-centric, we should limit submissions to only English and Spanish. So I did some research into language detection, found a service with an API that looked promising, and did an initial implementation. However, for technical reasons that I never did uncover, that API was not working on our hosting setup (even though it worked on my local system!).
We then changed our focus from language detection to the fact that our current problem was almost entirely due to submissions made in certain scripts (specific human writing systems).
I found that regular expressions in PHP support Unicode character properties. Those property codes include values that designate the Chinese script (which includes both traditional and simplified characters) and the Korean script.
We decided to take a zero-tolerance approach. A single Chinese or Korean character causes the user submission not to validate. So far, implementing this has drastically reduced the amount of inappropriate content and comments.
Here's the code, which is called by our Drupal form validation functions.
/**
* Check whether a string contains any characters from a
* banned script, such as Chinese or Korean.
*
* @param string $text
* The piece of text to be checked.
*
* @return TRUE | FALSE
* Returns TRUE if any Chinese or Korean characters detected.
* Otherwise, returns FALSE.
*/
function _contains_banned_scripts($text) {
$unicode_modifier = 'u';
// Detect whether there are any Chinese characters.
$chinese_regex = '\p{Han}+';
$preg_regex = '/' . $chinese_regex . '/' . $unicode_modifier;
$chinese_found = preg_match($preg_regex, $text);
if ($chinese_found == 1) {
return TRUE;
}
// Detect whether there are any Korean characters.
$korean_regex = '\p{Hangul}+';
$preg_regex = '/' . $korean_regex . '/' . $unicode_modifier;
$korean_found = preg_match($preg_regex, $text);
if ($korean_found == 1) {
return TRUE;
}
return FALSE;
}
Sources:
Forum spam
https://en.wikipedia.org/wiki/Forum_spam
Unicode Regular Expressions
http://www.regular-expressions.info/unicode.html
Unicode Character Properties
https://secure.php.net/manual/en/regexp.reference.unicode.php
Feb 24, 2016
An Exercise in Pair Programming
I've spent many years as a software developer working in contented solitude: wrapping my mind around spaghetti code; stepping through a debugger; implementing an algorithm. I'm an introvert, so I'm comfortable being alone.
The past few years, though, I've felt pulled to collaborate more with others and to participate more actively as a member of a team. I've done this by communicating with end users, clients, and non-technical team members as well as doing informal mentoring and teaching.
I've known about pair programming for some time but have never practiced it. So when it turned out that one of my colleagues and I were both learning jQuery and JavaScript, I suggested we do an exercise together via pair programming.
Luis and I have pretty different backgrounds. He is a college student majoring in computer engineering. I'm a very seasoned programmer who later moved into web technologies and have re-entered that field after a long hiatus.
We built a client-side game for a person to scramble and then solve the 15-puzzle (source code in GitHub repo ). This was done over a few weeks during off hours when we were able to schedule snippets of time together.
(Update: play the puzzle )
It went really well, and I want to jot down a few thoughts about what enabled it to be a positive, constructive experience.
* We already had a good working relationship.
Luis and I are both part-time developers at a digital consulting agency. We had worked together before and felt personally compatible, so much so that we've made a point of going into the office on the same days so that we can readily consult with each other.
* We did it as part of a learning exercise.
This was done not as part of our paid work but of our individual learning endeavors. Luis has been using one of the online services, I've been going through a traditional book. This meant that not much was at stake in terms of producing results, which reduced the potential stress level considerably.
* Neither of us dominated.
Although I am far senior in terms of general software experience, I am a novice with respect to JavaScript and jQuery. And most recently, my career path has been a pretty choppy one with a lot of non-technical roads taken and committed to. These factors together with a strong preference to collaborate rather than compete means that I simply treat Luis as a peer.
* We were open to each other's criticism and suggestions.
If you can sufficiently suspend your own preferences, habits, and beliefs about what's "right", it's possible to learn a lot from another developer. Each of us was willing to try coding differently. This ran the gamut from how to do indentation to what data structures to use for the underlying model.
Small example: in jQuery there is consistent and frequent use of anonymous functions that are passed as parameters. Luis suggested giving names to those functions even though the names are never used. Although this appears to go against the most common coding practice, I ended up liking the additional clarity and expression of intent that such unnecessary function names provide.
* Debugging went much faster and more efficiently.
Case in point. Luis realized that certain event handlers were not working because they no longer existed -- the DOM elements they were attached to were being removed and then some of those elements were re-created as replacements, minus their handlers. Stuff like that is obvious when someone else points it out!
For my part, I correctly suspected that a recursive use of a particular jQuery selector was causing the failure of an iterator to affect all intended objects. It's a subtle bug that I still don't understand the root cause of, but my background in programming languages led to a good guess.
* Both of us have an academic background.
This allowed me, for example, to jump into explaining terminology such as "operator precedence" and "operator associativity" without Luis' eyes glazing over. He not only has the intellectual chops, he gets it that taking the time to learn basic concepts and acquire fundamental understanding is worth it.
Doing this exercise has given me momentum and encouragement to look for opportunities to do more pair programming and to expand further my own envelope. Nice!
Resources:
Pair Programming
http://guide.agilealliance.org/guide/pairing.html
Pair Programming Considered Harmful?
http://techcrunch.com/2012/03/03/pair-programming-considered-harmful/
Pair Programming vs. Code Reviews
https://blog.codinghorror.com/pair-programming-vs-code-reviews/
The past few years, though, I've felt pulled to collaborate more with others and to participate more actively as a member of a team. I've done this by communicating with end users, clients, and non-technical team members as well as doing informal mentoring and teaching.
I've known about pair programming for some time but have never practiced it. So when it turned out that one of my colleagues and I were both learning jQuery and JavaScript, I suggested we do an exercise together via pair programming.
Luis and I have pretty different backgrounds. He is a college student majoring in computer engineering. I'm a very seasoned programmer who later moved into web technologies and have re-entered that field after a long hiatus.
We built a client-side game for a person to scramble and then solve the 15-puzzle (source code in GitHub repo ). This was done over a few weeks during off hours when we were able to schedule snippets of time together.
(Update: play the puzzle )
It went really well, and I want to jot down a few thoughts about what enabled it to be a positive, constructive experience.
* We already had a good working relationship.
Luis and I are both part-time developers at a digital consulting agency. We had worked together before and felt personally compatible, so much so that we've made a point of going into the office on the same days so that we can readily consult with each other.
* We did it as part of a learning exercise.
This was done not as part of our paid work but of our individual learning endeavors. Luis has been using one of the online services, I've been going through a traditional book. This meant that not much was at stake in terms of producing results, which reduced the potential stress level considerably.
* Neither of us dominated.
Although I am far senior in terms of general software experience, I am a novice with respect to JavaScript and jQuery. And most recently, my career path has been a pretty choppy one with a lot of non-technical roads taken and committed to. These factors together with a strong preference to collaborate rather than compete means that I simply treat Luis as a peer.
* We were open to each other's criticism and suggestions.
If you can sufficiently suspend your own preferences, habits, and beliefs about what's "right", it's possible to learn a lot from another developer. Each of us was willing to try coding differently. This ran the gamut from how to do indentation to what data structures to use for the underlying model.
Small example: in jQuery there is consistent and frequent use of anonymous functions that are passed as parameters. Luis suggested giving names to those functions even though the names are never used. Although this appears to go against the most common coding practice, I ended up liking the additional clarity and expression of intent that such unnecessary function names provide.
* Debugging went much faster and more efficiently.
Case in point. Luis realized that certain event handlers were not working because they no longer existed -- the DOM elements they were attached to were being removed and then some of those elements were re-created as replacements, minus their handlers. Stuff like that is obvious when someone else points it out!
For my part, I correctly suspected that a recursive use of a particular jQuery selector was causing the failure of an iterator to affect all intended objects. It's a subtle bug that I still don't understand the root cause of, but my background in programming languages led to a good guess.
* Both of us have an academic background.
This allowed me, for example, to jump into explaining terminology such as "operator precedence" and "operator associativity" without Luis' eyes glazing over. He not only has the intellectual chops, he gets it that taking the time to learn basic concepts and acquire fundamental understanding is worth it.
Doing this exercise has given me momentum and encouragement to look for opportunities to do more pair programming and to expand further my own envelope. Nice!
Resources:
Pair Programming
http://guide.agilealliance.org/guide/pairing.html
Pair Programming Considered Harmful?
http://techcrunch.com/2012/03/03/pair-programming-considered-harmful/
Pair Programming vs. Code Reviews
https://blog.codinghorror.com/pair-programming-vs-code-reviews/
Feb 1, 2016
Unwanted Side Effect of Reducing Permissions on Drupal Text Formats
On a Drupal 7 site, we were getting malicious comments that were triggering undesired page redirects, as I wrote about in Malicious page redirects and Drupal's Filtered-HTML text format.
One cause of those redirects was malicious html using the <meta> tag. This tag was allowed because Authenticated Users had access to the Full HTML text format. So we decided to limit use of Full HTML only to Administrators.
Since then, I have learned that this caused an undesirable side-effect. The symptom was that when a user tried to edit content that they had previously created, in the text area for doing the editing they got the following message.
This field has been disabled because you do not have sufficient permissions to edit it
Very puzzling! Why would a user not be able to edit their own content that they had created?
It turns out that when text content is created, the text format in effect at that time is stored with the content and is applied when editing is done later.
If you limit the roles that have access to a particular text format, then text content created earlier by users who have that role may no longer be editable by those users. They are blocked from making any changes.
On our site, this is not a serious problem. Text content is almost always written and submitted once and not revised further.
* * *
However, a site with content that is actively edited is going to run into a wall. There are a couple of things you might be able to do.
If the number of content items is small, then while logged in as an Administrator or some other role with sufficient permissions, edit each item so that it uses a Text Format that the author has permission to use. In our case, it would have been from Full HTML to Filtered HTML.
If you have a lot of content and you're dangerous enough to hack around directly with the database, then a database global change will also work.
For example, in our case, we have two tables that are relevant. Executing these SQL commands changes the values.
update field_data_body
set body_format = 'filtered_html'
where body_format = 'full_html';
update field_revision_body
set body_format = 'filtered_html'
where body_format = 'full_html';
(And don't forget to clear cache!)
Thanks to David Needham for a particularly pithy and helpful comment in the second article cited below.
Sources:
Signature box for authenticated users - This field has been disabled because you do not have sufficient permissions to edit it
https://www.drupal.org/node/1034064
This field has been disabled because you do not have sufficient permissions to edit it
https://www.drupal.org/node/1064168
One cause of those redirects was malicious html using the <meta> tag. This tag was allowed because Authenticated Users had access to the Full HTML text format. So we decided to limit use of Full HTML only to Administrators.
Since then, I have learned that this caused an undesirable side-effect. The symptom was that when a user tried to edit content that they had previously created, in the text area for doing the editing they got the following message.
This field has been disabled because you do not have sufficient permissions to edit it
Very puzzling! Why would a user not be able to edit their own content that they had created?
It turns out that when text content is created, the text format in effect at that time is stored with the content and is applied when editing is done later.
If you limit the roles that have access to a particular text format, then text content created earlier by users who have that role may no longer be editable by those users. They are blocked from making any changes.
On our site, this is not a serious problem. Text content is almost always written and submitted once and not revised further.
* * *
However, a site with content that is actively edited is going to run into a wall. There are a couple of things you might be able to do.
If the number of content items is small, then while logged in as an Administrator or some other role with sufficient permissions, edit each item so that it uses a Text Format that the author has permission to use. In our case, it would have been from Full HTML to Filtered HTML.
If you have a lot of content and you're dangerous enough to hack around directly with the database, then a database global change will also work.
For example, in our case, we have two tables that are relevant. Executing these SQL commands changes the values.
update field_data_body
set body_format = 'filtered_html'
where body_format = 'full_html';
update field_revision_body
set body_format = 'filtered_html'
where body_format = 'full_html';
(And don't forget to clear cache!)
Thanks to David Needham for a particularly pithy and helpful comment in the second article cited below.
Sources:
Signature box for authenticated users - This field has been disabled because you do not have sufficient permissions to edit it
https://www.drupal.org/node/1034064
This field has been disabled because you do not have sufficient permissions to edit it
https://www.drupal.org/node/1064168
Jan 15, 2016
jQuery.get() Ajax call fails on localhost url
While going through the book Learning jQuery, 4th Edition, I've been doing the exercises by placing and editing code files in a local directory on the desktop and opening index.html as a file in Firefox.
This worked fine until I had to code an Ajax call that referred to a php file. This required the use of a web server, so I created a subdirectory under the local server root and copied the php file into the subdir.
Keeping all other code on the desktop, I then edited the Ajax call so that it requested from the url
http://localhost/learning-jquery/e.php
The call silently failed and returned nothing. Console windows would show nothing, no errors.
Eventually, I found out about the same-origin policy. In brief, this policy means that by default JavaScript is prevented from making requests across domain boundaries, where same-origin means that the protocol, hostname, and port number must be identical.
So I moved all code files into the subdir under the server root. Working from there rather than the desktop, the Ajax calls then succeeded.
(And I changed the requested url back to just e.php)
***
But what about carrying out cross-origin access? One possible way is to configure the server to allow it. For Apache on my Linux system, I added the following directive in /etc/apache2/apache2.conf
<Directory /var/www/html/learning-jquery>
Header set Access-Control-Allow-Origin "*"
</Directory>
However, checking this configuration change with the command
# apachectl -t
resulted in
AH00526: Syntax error on line 205 of /etc/apache2/apache2.conf:
Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration
Action '-t' failed.
The Apache error log may have more information.
It turns out that I also had to enable a module named headers
# a2enmod headers
Then restart the server. That enabled cross-origin access.
Not yet satisfied, I was wondering if the wildcard "*" in the directive could be replaced by something more restrictive. For the way I'm developing locally, the following also works because opening a file via the file:// protocol results in an origin of null.
<Directory /var/www/html/learning-jquery>
Header set Access-Control-Allow-Origin "null"
</Directory>
***
The fact that the calls had failed silently was very troubling. One way to avoid this is to register a global Ajax error handler by calling ajaxError(). Here's a handler that simply puts up an alert box. (I'm using jQuery 1.9)
$(document).ready(
function setAjaxErrorHandler() {
$(document).ajaxError(
function alertError(event, jqxhr, settings, thrownErr) {
alert('Ajax error handler: ' + jqxhr.status
+ ' ' + jqxhr.statusText);
}
);
}
);
For the cross-origin error, this puts up the message "Ajax error handler: 0 error", which is not exactly super helpful but better than nothing.
Another way is to attach an error handler to the particular request by using the fail() method. For example,
$.get('http://localhost/learning-jquery/e.php', { ... },
function (data) { ... })
.fail(function (jqxhr) {
alert('GET error: ' + jqxhr.status + ' '
+ jqxhr.statusText);
});
Sources:
jQuery.get()
https://api.jquery.com/jQuery.get/
Same-origin policy
https://en.wikipedia.org/wiki/Same-origin_policy
Why is CORS important?
http://enable-cors.org/
CORS on Apache
http://enable-cors.org/server_apache.html
Configure Apache To Accept Cross-Site XMLHttpRequests on Ubuntu
https://harthur.wordpress.com/2009/10/15/configure-apache-to-accept-cross-site-xmlhttprequests-on-ubuntu/
This worked fine until I had to code an Ajax call that referred to a php file. This required the use of a web server, so I created a subdirectory under the local server root and copied the php file into the subdir.
Keeping all other code on the desktop, I then edited the Ajax call so that it requested from the url
http://localhost/learning-jquery/e.php
The call silently failed and returned nothing. Console windows would show nothing, no errors.
Eventually, I found out about the same-origin policy. In brief, this policy means that by default JavaScript is prevented from making requests across domain boundaries, where same-origin means that the protocol, hostname, and port number must be identical.
So I moved all code files into the subdir under the server root. Working from there rather than the desktop, the Ajax calls then succeeded.
(And I changed the requested url back to just e.php)
***
But what about carrying out cross-origin access? One possible way is to configure the server to allow it. For Apache on my Linux system, I added the following directive in /etc/apache2/apache2.conf
<Directory /var/www/html/learning-jquery>
Header set Access-Control-Allow-Origin "*"
</Directory>
However, checking this configuration change with the command
# apachectl -t
resulted in
AH00526: Syntax error on line 205 of /etc/apache2/apache2.conf:
Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration
Action '-t' failed.
The Apache error log may have more information.
It turns out that I also had to enable a module named headers
# a2enmod headers
Then restart the server. That enabled cross-origin access.
Not yet satisfied, I was wondering if the wildcard "*" in the directive could be replaced by something more restrictive. For the way I'm developing locally, the following also works because opening a file via the file:// protocol results in an origin of null.
<Directory /var/www/html/learning-jquery>
Header set Access-Control-Allow-Origin "null"
</Directory>
***
The fact that the calls had failed silently was very troubling. One way to avoid this is to register a global Ajax error handler by calling ajaxError(). Here's a handler that simply puts up an alert box. (I'm using jQuery 1.9)
$(document).ready(
function setAjaxErrorHandler() {
$(document).ajaxError(
function alertError(event, jqxhr, settings, thrownErr) {
alert('Ajax error handler: ' + jqxhr.status
+ ' ' + jqxhr.statusText);
}
);
}
);
For the cross-origin error, this puts up the message "Ajax error handler: 0 error", which is not exactly super helpful but better than nothing.
Another way is to attach an error handler to the particular request by using the fail() method. For example,
$.get('http://localhost/learning-jquery/e.php', { ... },
function (data) { ... })
.fail(function (jqxhr) {
alert('GET error: ' + jqxhr.status + ' '
+ jqxhr.statusText);
});
Sources:
jQuery.get()
https://api.jquery.com/jQuery.get/
Same-origin policy
https://en.wikipedia.org/wiki/Same-origin_policy
Why is CORS important?
http://enable-cors.org/
CORS on Apache
http://enable-cors.org/server_apache.html
Configure Apache To Accept Cross-Site XMLHttpRequests on Ubuntu
https://harthur.wordpress.com/2009/10/15/configure-apache-to-accept-cross-site-xmlhttprequests-on-ubuntu/
Jan 7, 2016
How to Extract Columns from a CSV file (without using a spreadsheet)
One of my colleagues who does data analysis had a CSV (comma-separated values) file that he could not open in Excel nor in any other spreadsheet program he tried.
Either due to its sheer filesize of 125MB, or its number of rows of more than 1,500,000, the programs would gag.
It turned out that for his purposes, he did not need all of the data, only a subset of the columns. Maybe extracting only what he needed into a smaller CSV would enable him to be able to work with it.
I had earlier read parts of the book The Linux Command Line, by William Shotts. I vaguely remembered mention of a utility to selectively pull fields out of a text file. That turned out to be the cut command.
Here's what the first few rows of the original data file looked like.
id,type,distance,userid,charityID,time,lat,lon
1003529743,walk,4.48342,1000545086,2166731,"2015-06-30 00:00:05",40.2501,-76.6714
1003529744,Run,4.087,1000402641,15048,"2015-06-30 00:00:21",45.5244,-89.7398
1003529745,run,2.631135,1000258381,61635018,"2015-06-30 00:00:23",41.6281,-87.193
1003529746,Bike,1.216703,1000505816,18010,"2015-06-30 00:00:24",43.0306,-78.7963
1003529747,walk,2.069664,1000015957,18010,"2015-06-30 00:00:25",39.2481,-76.5165
1003529748,Bike,6.174,1000126350,18010,"2015-06-30 00:00:25",29.5913,-82.4298
1003529749,run,1.47652,1000542869,92985044,"2015-06-30 00:00:26",40.7115,-89.4287
Only the type and userid columns were actually required. Using the following command I was able to generate another CSV with just those two fields.
$ cut -f 2,4 -d, july.csv > type-userid.csv
The -f option specifies which fields to extract. In this case, its the 2nd and the 4th fields. The -d option is the field delimiting character, which in our case is the comma. It defaults to tabs.
july.csv is the input file, type-userid.csv captures the standard-out.
The first few rows of the resulting file were
type,userid
walk,1000545086
Run,1000402641
run,1000258381
Bike,1000505816
walk,1000015957
Bike,1000126350
run,1000542869
And the filesize was reduced to about 24MB, which was usable and much more manageable.
(However, depending on which spreadsheet app you are using, the row count might exceed the limit. For example, both Excel 2007 and LibreOffice 4.2 Calc can handle a maximum of 1,048,576 rows.)
***
The cut command works blazingly fast. It only took about a second to process the input file. After I handed off the outputted file, I later realized that I might have been able to save my colleague some tedium and waiting time by applying other CLI text-processing commands as well, such as sort, uniq, and wc, depending on what he wanted to do.
Source:
The Linux Command Line, by William Shotts
http://linuxcommand.org/tlcl.php
(You can buy the paper book or download the pdf for free)
Either due to its sheer filesize of 125MB, or its number of rows of more than 1,500,000, the programs would gag.
It turned out that for his purposes, he did not need all of the data, only a subset of the columns. Maybe extracting only what he needed into a smaller CSV would enable him to be able to work with it.
I had earlier read parts of the book The Linux Command Line, by William Shotts. I vaguely remembered mention of a utility to selectively pull fields out of a text file. That turned out to be the cut command.
Here's what the first few rows of the original data file looked like.
id,type,distance,userid,charityID,time,lat,lon
1003529743,walk,4.48342,1000545086,2166731,"2015-06-30 00:00:05",40.2501,-76.6714
1003529744,Run,4.087,1000402641,15048,"2015-06-30 00:00:21",45.5244,-89.7398
1003529745,run,2.631135,1000258381,61635018,"2015-06-30 00:00:23",41.6281,-87.193
1003529746,Bike,1.216703,1000505816,18010,"2015-06-30 00:00:24",43.0306,-78.7963
1003529747,walk,2.069664,1000015957,18010,"2015-06-30 00:00:25",39.2481,-76.5165
1003529748,Bike,6.174,1000126350,18010,"2015-06-30 00:00:25",29.5913,-82.4298
1003529749,run,1.47652,1000542869,92985044,"2015-06-30 00:00:26",40.7115,-89.4287
Only the type and userid columns were actually required. Using the following command I was able to generate another CSV with just those two fields.
$ cut -f 2,4 -d, july.csv > type-userid.csv
The -f option specifies which fields to extract. In this case, its the 2nd and the 4th fields. The -d option is the field delimiting character, which in our case is the comma. It defaults to tabs.
july.csv is the input file, type-userid.csv captures the standard-out.
The first few rows of the resulting file were
type,userid
walk,1000545086
Run,1000402641
run,1000258381
Bike,1000505816
walk,1000015957
Bike,1000126350
run,1000542869
And the filesize was reduced to about 24MB, which was usable and much more manageable.
(However, depending on which spreadsheet app you are using, the row count might exceed the limit. For example, both Excel 2007 and LibreOffice 4.2 Calc can handle a maximum of 1,048,576 rows.)
***
The cut command works blazingly fast. It only took about a second to process the input file. After I handed off the outputted file, I later realized that I might have been able to save my colleague some tedium and waiting time by applying other CLI text-processing commands as well, such as sort, uniq, and wc, depending on what he wanted to do.
Source:
The Linux Command Line, by William Shotts
http://linuxcommand.org/tlcl.php
(You can buy the paper book or download the pdf for free)
Dec 26, 2015
Drupal 8, Beta 15 --> RC 1: Eliminating use of checkPlain( )
Since SafeMarkup::checkPlain() is now deprecated and possibly intended to be removed along with other SafeMarkup methods, I decided to address this in migrating to RC 1.
In the common use case where D8's Twig templates are used for output, you can dispense with checkPlain() entirely and rely on Twig's autoescaping. That turned out to apply to our simple use of forms to input and render a few text values.
In the original code, checkPlain() was called both to process user-entered values as well as to sanitize those values before they were placed into render arrays. When I typed <script> into a form field, it was incorrectly escaped twice and displayed as <script>
Removing those calls works fine and conforms to what is considered good practice for D8. Potentially unsafe markup is stored as is in the database, but Twig converts it before it is sent to the browser.
The article SafeMarkup methods are removed is extremely useful in how it breaks down the different use cases for checkPlain() and what needs to be done differently for each in D8 in order to prevent unsafe markup from being rendered.
Besides Twig templates, the other use cases are:
- Text placed into a render array by using the #plain_text key
- A mixture of escaped markup with markup not to be escaped
- Non-HTML responses, eg. JSON
The discussion also applies to the check_plain() function in Drupal 7, for which checkPlain() was a replacement. If you're starting out with a conversion from D7, the article is a must read.
Sources:
SafeMarkup::set(), SafeMarkup::checkPlain(), and other methods are removed from Drupal 8 core
https://groups.drupal.org/node/478558
SafeMarkup methods are removed
https://www.drupal.org/node/2549395
Twig autoescape enabled and text sanitization APIs updated
https://www.drupal.org/node/2296163
In the common use case where D8's Twig templates are used for output, you can dispense with checkPlain() entirely and rely on Twig's autoescaping. That turned out to apply to our simple use of forms to input and render a few text values.
In the original code, checkPlain() was called both to process user-entered values as well as to sanitize those values before they were placed into render arrays. When I typed <script> into a form field, it was incorrectly escaped twice and displayed as <script>
Removing those calls works fine and conforms to what is considered good practice for D8. Potentially unsafe markup is stored as is in the database, but Twig converts it before it is sent to the browser.
The article SafeMarkup methods are removed is extremely useful in how it breaks down the different use cases for checkPlain() and what needs to be done differently for each in D8 in order to prevent unsafe markup from being rendered.
Besides Twig templates, the other use cases are:
- Text placed into a render array by using the #plain_text key
- A mixture of escaped markup with markup not to be escaped
- Non-HTML responses, eg. JSON
The discussion also applies to the check_plain() function in Drupal 7, for which checkPlain() was a replacement. If you're starting out with a conversion from D7, the article is a must read.
Sources:
SafeMarkup::set(), SafeMarkup::checkPlain(), and other methods are removed from Drupal 8 core
https://groups.drupal.org/node/478558
SafeMarkup methods are removed
https://www.drupal.org/node/2549395
Twig autoescape enabled and text sanitization APIs updated
https://www.drupal.org/node/2296163
Subscribe to:
Posts (Atom)