While converting automated tests to class WebTestBase in Drupal 8, several of the assertions were failing. This was happening in spite of the fact that none of the theme templates, element ids, or other relevant code had changed.
These assertions had been implemented with assertRaw(), which does literal string matching against the raw HTML of the response pages. It turns out that in D8, some form elements may be generated differently even though the final rendering is pretty much the same.
As a slightly simplified example, here's a tag that is output by the Drupal 7 version of the module.
<input disabled="disabled" type="text" id="edit-optimizely-project-code" name="optimizely_project_code" value="Undefined" size="30" />
But in D8, the corresponding tag is
<input aria-describedby="edit-optimizely-project-code--description" disabled="disabled" type="text" id="edit-optimizely-project-code" name="optimizely_project_code" value="Undefined" size="30" aria-required="true" />
Obviously, these two strings don't match.
The intent of the assertion is to check that the particular input tag exists, that it is disabled, and that it has a value of "Undefined". The other attributes are irrelevant.
So I re-wrote the assertion using assertPattern() instead, which takes a regular expression as its first parameter.
$this->assertPattern(
':<input .*? disabled="disabled" .*? id="edit-optimizely-project-code" .*? value="Undefined" .*? />:', ...);
This is not perfect. For one thing, it still relies on the three attributes being in a certain order. It also has some extra space characters that could break the match, but I'm leaving them here to make the regex more readable.
However, I believe implementing with assertPattern() makes the test more robust in the face of future changes as well as making the code better self-documenting.
Sources:
Assertions
https://www.drupal.org/node/265828
abstract class WebTestBase
https://api.drupal.org/api/drupal/core!modules!simpletest!src!WebTestBase.php/class/WebTestBase/8
Does testing for a pattern open up the possibility of the test returning a false positive? Would it not be better that the test breaks if the markup changes thus confirming the change as been applied and the test needs adjusting? Your solution sounds like a good approach for an application but perhaps not for a test?
ReplyDeleteThat's a very good point about the possibility of false positives. It can be hard to write a regular expression that is exactly correct.
DeleteUnlike the rather simplistic one that is in this post, here's one that I actually committed.
:<input( .*? | )disabled="disabled"( .*? | )id="edit-optimizely-project-title"( .*? | )value="Default"( .*?)?/>:
Your comment prompted me to take another close look at this regex. I can see now that it could match across multiple html tags and therefore be a false positive.
On the other hand, it seems extremely unlikely for that to happen.
It bothers me that markup changes that are out of our control and not relevant to the Optimizely module can break the automated tests and make it necessary to change our code.
DeleteIn the current scenario, we are converting from one major version of Drupal to another, which is not going to happen often.
But what about minor releases of Drupal? Of if the user switches to a different theme? I don't know the answers.
I revisited the regex I posted in my previous comment and made it even less likely to produce a false positive. This one is intended to match only within a single HTML tag.
Delete:<input( [^>]*? | )disabled="disabled"( [^>]*? | )id="edit-optimizely-project-title"( [^>]*? | )value="Default"( [^>]*?)?/>:
I really did not do enough testing when I wrote the original expression.
But I found this online regex evaluator, which is a huge, huge help. It says it's "for Perl", but for the kind of basic regex that I wrote, it works fine.
http://www.regexplanet.com/advanced/perl/index.html