LD.
Music, software, life… and stuff.
[ Twitter ] [ GitHub ] [ Linked In ]
In response to Rob’s post on functional testing.
Rob, a Geb committer and all around cool dude, recently posted some thoughts on function testing.
The following is what I intended to be a comment on Rob’s blog, but it became too long so I decided to post it here. You’ll notice that the language is a bit weird as in some places I am addressing Rob directly. It should make sense (no guarantees) if you pretend you are reading it as a comment on Rob’s blog.
This is not a flame war, though that would be fun. Rob’s a friend of mine and a respected colleague. This is just a discussion and exchange of ideas.
Without further ado, the comment…
I’ll try to be as objective as I can, but obviously as someone who is invested in Geb I come with a bit of a bias.
You seem to be mixing up several different issues, which is understandable as the post seems born out of frustration.
For my mind there are three discrete parts:
- The development environment (the language/framework/constructs you use to express the test)
- The execution environment
- The debugging environment
They lines blur, but I think it’s helpful to think about each part separately. Geb itself only lives in the first arena strictly speaking. How it works in the execution and debugging environment are largely aspects of the associated tooling, with the exception of static diagnostic information.
So a couple of points:
“I agree that as much testing as possible should be done at as low a level as possible”
What’s usually missing from this sentiment is the why. Too many people take this stance because testing at a low level is easier. This, in isolation, is not the right way to think about it. The end goal is always working software, not easy tests. Easy tests work towards working software by making it more likely that tests will be created in the first place and maintained. They are low cost/low value. So for me, low level testing is fine until the accuracy compromist becomes too high. If it doesn’t create a lot of confidence that the system will work, either because it assumes to much or is not backed up by higher level tests, then the developer is fooling themselves. Passing tests is not the end goal, working software is.
It’s with that in mind that I usually say to people that we need to “suck it up” to a certain extent with functional tests. You should identify if there is value in having automated tests that verify the end to end behaviour of your system, get a feel for what that value is and act accordingly. Then you can make cost/benefit judgements on how much effort to put into functional testing. Avoiding it because it’s not as easy as unit testing while not taking the value into account is irresponsible, in my opinion. However, we definitely should harness the frustration and desire for simplicity to make it easier.
“Even assuming you can optimise so that the application is running and you can run the test from an IDE then Selenium has to start up a browser instance, queries execute, views need to render, etc”
Selenium doesn’t have to start up a browser instance, it’s possible to use an running browser instance. Also, on modern development machines we should be talking about seconds at most to run queries and render views etc.
As for the Grails specific issues, you’re going to face the slow application startup when working with Grails no matter what your development environment. In theory, there’s no difference between different development environments here, but the difference comes in in execution environments. However, it’s possible to create an execution environment for Geb tests (run in IDE against a running application) that is equivalent to the Selenium IDE experience.
“Geb’s output when a waitFor condition fails is just Condition did not pass in x seconds.”
This is indeed painful. Even with implicit power asserts it doesn’t get much better. Providing static diagnostic information for functional tests (i.e. non dynamic like an interactive debugging session) is one of the unsolved problems in this area. I’m not aware of any tool that has a good approach to this. With Geb our approach is to just dump the browser state, but that information is generally not rich enough. Hopefully someone comes up with a good solution.
“Selenese is by no means great in this regard (a humble Condition timed out isn’t much help) but at least you can step back with the Selenium IDE and watch the damn thing not working much more easily.”
I think much more easily is a stretch. You can do the same thing with a Geb/WebDriver test. The only difference being is that the test method is your smallest level of granularity for re-running a part of the test. That’s a clear advantage that Selenium IDE has; the ability to run a selected portion of the story at any granularity. It seems like this would break down for most stories though as they require context that needs to be run as well.
“The most productive I’ve ever been when writing functional tests has been when using Selenium IDE.”
(and the rest of this paragraph and the next)
There’s only one point you make here that can’t be replicated with Geb/WebDriver in an IDE: selecting a portion of the test at any granularity and just running that. Everything else is also possible in an IDE, and is the exact same process you would use to debug other code. These are all points on the execution/debugging environment not the development environment.
“Despite these significant failings writing tests in Selenium IDE is very effective. Maintaining a suite of such tests is another matter. Working on a long-running project the failings of Selenese tests start to increase logarithmically.”
This is the point for me. Experience has shown that the majority of the cost of a functional test suite comes in maintenance, not initial development. Therefore the sensible thing to do is to optimise for this first and development speed second. Maintenance should be foremost in the developer’s mind when developing functional test. The PageObject pattern and inline test data are the two best techniquest that I am aware of that help here.
“I’m convinced that the goal of writing tests in the same language as the application is a pretty vapid one. Working inside one’s comfort zone is all very well but too many times I’ve seen things tested using Selenium or Geb that would be better tested with a unit-level JavaScript test.”
It’s not necessarily about using the same language, it’s about integration and sharing data structures. Equating functional testing to black box testing is incorrect I think. I have no problem with stimulating the system as a user, but poking inside to verify internal state. To me, that’s more productive and accurate.
JavaScript unit tests aren’t a replacement for functional tests. You shouldn’t be using functional tests to unit test JavaScript, but having JavaScript unit tests doesn’t relieve you from having functional tests. It can mean you need less though. If a test doesn’t emulate what a user does (including the user environment) to a reasonable degree of accuracy it is not a functional test.
“As a Grails developer I’ve looked enviously at how fast tests run in Rails apps but that’s nothing compared to watching a couple of hundred Jasmine tests run in under a second.”
I think you’re comparing apples and oranges here. I’d have little faith in a system that only has JavaScript unit tests.
“I was at one time convinced that the ability to use loops and conditional statements in Groovy made it a more suitable testing language than Selenese but honestly, how often are such constructs really required for tests?”
In my experience, these constructs are more useful when modelling the concept with appropriate abstractions. As interaction with web pages becomes richer (gestures, dragging etc.), I think we’ll see it become more useful there. That is, one logical interaction may have looping/branching logic internally but we want to model it as one abstraction.
“Building Geb content definitions with deeply nested layers of Module types is time consuming & difficult.”
I agree. However, when you put the effort in and do this right you get maintainable tests plus a wealth of predefined abstractions to draw on in future tests. That is, high cost/high value. With my Geb hat on, I think we have the best story in this area in terms of provided constructs. What we lack is the shared knowledge and experience to deploy these constructs effectively. I think every tool that tries to do this has the same problem though. In a word, inexperience.
“I can’t help thinking the page object model approach is coming at the problem from the wrong angle. Instead of abstracting the UI shouldn’t we be abstracting the behaviour?”
I think you need both. You can’t do away with abstracting the UI.
Geb’s in a tight spot here in that it has no execution model. This makes it awkward to introduce behaviour abstractions as we’d have to wrestle the execution control away from the execution framework we are integrating with. Ultimately, this abstraction belongs in JUnit/Spock/whatever. Spock is missing this capability, and that’s a known problem.
“The most impressive Selenium extension I’ve seen is Steve Cresswell’s Natural Language Extensions that layers something like JBehave’s feature definition language on top of Selenese.”
Not being a Selenese expert, this seems like it would give you the the behaviour abstractions you desire but you’d have to either largely abandon UI abstraction or maintain that separately.
I’d love to see someone start a project adapting something like Fitnesse or some other natural language based tool to Geb. You’d get the same behaviour abstraction benefits, but get to use Geb’s page modelling to maintain the UI abstractions. Hopefully someone starts this.
As for cucumber, if you want natural language authoring I think it’s the choice. There are posts out there about people layering it on top of Geb.
I’m very skeptical of FuncUnit, but not well educated about it. It’s not clear to me how it fits into an automated build and it seems like it would have a very high mental context switching cost. You also give up on data structure sharing entirely.
That’s it.
So the take home for me is that we need to spend more effort on the execution and debugging environments for Geb. Perhaps this will be the focus of the 0.7.x series which will be starting soon.
Posted: Nov 25th, 2011 @ 12:49 am
Geb Modules and isDisplayed()
A question recently came up on the Geb user mailing list about checking to see if a module is displayed on a page. It turns out the concept of displayed could do with some exploration.
Say we have the following html…
<p class="a" style="display:none"/>
<p class="b"/>
The following assertions will pass…
def a = $("p.a")
def b = $("p.b")
def c = $("p.c")
assert a && !a.displayed
assert b && b.displayed
assert !c && !c.displayed
So isDisplayed() means that the navigator matches some content and the first thing that it has matched is actually visible on the page. $("p.c").displayed returns false because it doesn’t match anything.
Navigator objects implement the Groovy Truth, so those assertions could be written as…
assert a.size() != 0 && !a.displayed
Modules also implement the Groovy Truth using the same test, so…
class ParagraphA extends Module {
static base = { $("p.a") }
}
class ParagraphB extends Module {
static base = { $("p.b") }
}
class ParagraphB extends Module {
static base = { $("p.c") }
}
Now assume we have moduleA, moduleB and moduleC vars that are instances of the corresponding modules, and we can write the following assertions…
assert moduleA && !moduleA.$().displayed
assert moduleB && moduleB.$().displayed
assert !moduleC && !moduleC.$().displayed
You can get the “base” of a module at anytime with «module».$(), just like you can do relative lookups (i.e. someModule.$(“p.d”))
Posted: Jun 16th, 2011 @ 9:56 am
Using Grails tags where you can’t use Grails tags
Sometimes in a Grails app, you need to call a tag where you can’t call a tag. This is usually indicative of a bad design in your application so do use the following with caution, but should you absolutely need to… the following is how you can.
In this example, we are going to add the ability to call tags to a filter to workaround GRAILS-6814. We want to call the createLink tag.
Note that GRAILS-6814 has been resolved for Grails 1.4, so this workaround for redirects to a different domain only applies to Grails 1.3 and earlier
import org.codehaus.groovy.grails.web.pages.GroovyPage
import org.springframework.web.context.request.RequestContextHolder
class HttpsOnlyFilter {
def gspTagLibraryLookup // need this autowired
def filters = {
redirectToHttps(controller: "*", action: "*") {
before = {
if (!request.secure && isSecurePage(controllerName, actionName)) {
def link = invokeTag("createLink", [
controller: controllerName,
action: actionName,
params: params,
base: "https://secure.domain.com"
]).toString()
response.status = 301
response.setHeader("Location", link)
false
}
}
}
}
def invokeTag(name, attrs = [:], body = null) {
def namespace
def tagName
if (name.contains(":")) {
def split = name.split(":", 2)
namespace = split[0]
tagName = split[1]
} else {
namespace = GroovyPage.DEFAULT_NAMESPACE
tagName = name
}
GroovyPage.captureTagOutput(gspTagLibraryLookup, namespace, tagName, attrs, body, RequestContextHolder.currentRequestAttributes())
}
}
The invokeTag method takes; the name of a tag, a map of attributes, and an option body closure and returns a StreamCharBuffer (so if you want the content as a String you new to toString() the return). Inside the body closure, you can write to the out stream just like in a tag implementation.
If the tag name given to invokeTag() doesn’t include a namespace, the default g namespace is used. To invoke a tag in a different namespace, you can use…
invokeTag("someNamespace:someTag", [param1: 1])
Remember, requiring tags outside of the view layer can be indicative of a design problem so do think long and hard about whether you really need to before using the above. Also, the above will only work in a request environment so if you try to use the above on a background thread were a fake request environment hasn’t been established it will blow up due to RequestContextHolder.currentRequestAttributes() returning null. If you need to setup a fake request environment, you can use GrailsWebUtil.bindMockWebRequest() to do so.
Posted: May 11th, 2011 @ 11:40 am
Always developing Grails apps with https and testing against the WAR
In response to a number of bugs creeping through to production due to classpath differences and ssl issues, I went about setting up my culprit grails app to always functionally test as a war and with https in both development and testing. Here’s how to do it.
It turns out that there are two bugs/issues with the current grails tomcat plugin that get in the way of this objective, so you’ll have to install the
1.3.7.1version of the tomcat plugin which fixes GRAILS-6688 and GRAILS-7488 for Grails 1.3.x (these issues will be fixed for Grails 1.4).
Always support https
This is quite simple, but perhaps a little tricky to work out. Simply put the following in your scripts/_Events.groovy…
eventParseArgumentsEnd = {
argsMap.https = true
}
That’s the only suitable hook that I have found that can be used to force https to be used all the time (i.e. run-app and test-app). It’s a little heavy, but I’m yet to find a situation where it causes an issue.
Always functionally testing with a WAR
Again, this is simple. The following goes in scripts/_Events.groovy…
eventAllTestsStart = {
testOptions.war = true
}
This is equivalent to always passing the -war argument to test-app.
Functional tests and SSL (trust)
Depending on how you are functionally testing (and you know you should be using Geb + Spock) you may have issues with the certificate that Grails will generate for you to serve https. Namely, you might need to go and configure trust settings in all of your functional test browsers etc. If you have multiple CI builds then you are going to have to do this for a lot of certs. Even worse, the cert that Grails generates is stored in the work dir so can be wiped out at any time.
You can minimise this pain by generating your own cert and having Grails use that when serving https instead of creating one on demand. To do this, you need the following two settings in BuildConfig.groovy…
grails.tomcat.keystorePath = "${basedir}/grails-app/conf/httpsKeystore"
grails.tomcat.keystorePassword = "secret"
You of course need to create this keystore with a cert in it, which you can do (in grails-app/conf) via…
keytool -genkey -alias localhost -dname CN=localhost,OU=Test,O=Test,C=US -keyalg RSA -validity 365 -storepass key -keystore httpsKeystore -storepass secret -keypass secret
You now only need to make sure the browser trusts this cert.
Why bother?
In my case, we were hitting some dependency issues that were due to the development and production classpaths being different so running functional tests against a war minimises this risk because the app is then run in a separate JVM without the Grails build time classpath, making it more production like.
The application that we were having problems with uses mixed http and https and is also reasonably AJAX heavy. Several issues had crept through with AJAX requests being made on secure pages to non secure pages (which will blow up due to the browser not being willing to compromise security by making the unsecure request).
These two relatively simple steps get you closer to a production environment if you don’t have the resourcing, manpower, will etc. to setup an elaborate QA environment for automated testing.
Posted: May 6th, 2011 @ 1:54 pm
Decorating Constructors with Groovy
I recently put out a new plugin for Grails that brings something like view models to Grails applications. An interesting feature of this plugin is that it supports direct object instantiation with implicit autowiring. You might think that Grails already support this, but it doesn’t. Currently, you can directly instantiate domain objects and get autowiring but you can’t write a constructor and use it here.
Groovy has had the ability to fiddle with a class’s constructors via EMC for a while.
class Thing {
def i
Thing() { i = 1 }
}
assert new Thing().i == 1
Thing.metaClass.constructor = { -> i = 2 }
assert new Thing().i == 2
Trying to augment existing constructors is a bit trickier though. Using the standard EMC API of setting a constructor property on the meta class, there is no way to access the constructor that is being replaced. To do this you have to dig a bit deeper.
For the view models plugin I needed to invoke the real constructor, then do some stuff to the new instance. My solution revolves around the registerInstanceMethod(groovy.lang.MetaMethod method) method of ExpandoMetaClass. After some digging through Groovy, I found that the technique above of setting a constructor property on the meta class boils down to just attaching an instance MetaMethod with a method name of <init> which is the name Groovy uses for constructors. So what we can do is subclass groovy.lang.MetaMethod to get our custom behaviour, which looks like this…
package grails.plugin.viewmodels.constructor;
import java.lang.InstantiationException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import groovy.lang.MetaMethod;
import org.codehaus.groovy.reflection.CachedClass;
import org.codehaus.groovy.reflection.ReflectionCache;
public class DecoratingMetaConstructor<T> extends MetaMethod {
final private Constructor<T> constructor;
final private ConstructionDecorator<T> decorator;
public DecoratingMetaConstructor(Constructor<T> constructor, ConstructionDecorator<T> decorator) {
super(decorator.getSignature(constructor));
this.constructor = constructor;
this.decorator = decorator;
}
public Constructor<T> getConstructor() {
return constructor;
}
public int getModifiers() {
return constructor.getModifiers();
}
public String getName() {
return "<init>";
}
public Class<T> getReturnType() {
return constructor.getDeclaringClass();
}
public CachedClass getDeclaringClass() {
return ReflectionCache.getCachedClass(constructor.getDeclaringClass());
}
public Object invoke(Object object, Object[] arguments) {
Object[] transformedArgs = decorator.transformArgs(arguments);
T instance = instantiate(transformedArgs);
decorator.decorate(instance, arguments, transformedArgs);
return instance;
}
protected T instantiate(Object[] arguments) {
try {
return decorator.transformConstructor(constructor).newInstance(arguments);
} catch (InstantiationException e) {
throw new RuntimeException("Failed to invoke constructor " + constructor + " with args: " + arguments, e);
} catch (IllegalAccessException e) {
throw new RuntimeException("Failed to invoke constructor " + constructor + " with args: " + arguments, e);
} catch (InvocationTargetException e) {
throw new RuntimeException("Failed to invoke constructor " + constructor + " with args: " + arguments, e);
}
}
}
A lot of that is just satisfying the MetaMethod methods. What’s of interest to us is that we take a constructor object, and a decorator which is an interface that we provide…
package grails.plugin.viewmodels.constructor;
import java.lang.reflect.Constructor;
public interface ConstructionDecorator<T> {
public Object[] transformArgs(Object[] args);
public void decorate(T instance, Object[] originalArgs, Object[] transformedArgs);
public Class<?>[] getSignature(Constructor<T> constructor);
public Constructor<T> transformConstructor(Constructor<T> constructor);
}
The decorator can supply a different set of args for the actual Constructor to those that were given to the invocation, supply a different Constructor instance to use at instantiation time and/or do something with the instance after it has been created.
We can use this mechanism to write a construction decorator that performs autowiring…
package grails.plugin.viewmodels.constructor;
import java.lang.reflect.Constructor;
import org.springframework.context.ApplicationContext;
public class AutowiringConstructionDecorator<T> extends ConstructionDecoratorSupport<T> {
final private Autowirer autowirer;
public AutowiringConstructionDecorator(ApplicationContext applicationContext) {
this(new Autowirer(applicationContext));
}
public AutowiringConstructionDecorator(Autowirer autowirer) {
this.autowirer = autowirer;
}
public void decorate(T instance, Object[] givenArgs, Object[] transformedArgs) {
autowirer.autowire(instance);
}
}
The Autowirer takes care of the actual autowiring details, which aren’t of much interest here (see the class here if you want).
So we can now take a class, and make all of it’s constructors implicitly perform autowiring by doing something like this…
def autowiringDecorator = new AutowiringConstructionDecorator(grailsApplication.mainContext)
for (constructor in MyThing.constructors) {
MyThing.metaClass.registerInstanceMethod(new DecoratingMetaConstructor(constructor, autowiringDecorator))
}
Interestingly, I was able to use the same decoration mechanism to enable hot reloading support by obtaining the newest version of the constructor from the class loader in the Grails application that has the newest version of the class…
package grails.plugin.viewmodels.constructor;
import java.lang.reflect.Constructor;
class ReloadCapableConstructionDecorator<T> extends ConstructionDecoratorSupport<T> {
private final ClassLoader classLoader;
public ReloadCapableConstructionDecorator(ClassLoader classLoader) {
this.classLoader = classLoader;
}
public Constructor<T> transformConstructor(Constructor<T> constructor) {
try {
Class<T> originalClassVersion = constructor.getDeclaringClass();
Class<T> newestClassVersion = (Class<T>)classLoader.loadClass(originalClassVersion.getName());
return newestClassVersion.getConstructor(constructor.getParameterTypes());
} catch (NoSuchMethodException e) {
return constructor;
} catch (ClassNotFoundException e) {
return constructor;
}
}
}
The details of why this is needed isn’t worth going into now, what’s important is the fact that you could do some rather interesting stuff here.
If you find you have the need to decorate existing constructors as opposed to completely replacing them, it’s probably worth taking a look at what’s in the view-models plugin and either stealing it verbatim or adapting to your own purposes.
The relavant code and gory details can be found here.
Posted: Mar 4th, 2011 @ 3:54 pm
Tags: #software #grails #groovy
