feat: RuleTester supports processor

[mysticatea]

Link to rendered RFC

Summary

This RFC makes RuleTester class supporting processor option.

Related Issues

• Update: RuleTester Improvements #25 (comment)

[ilyavolodin]

I'm not convinced that this is needed either. Unit tests are supposed to test smallest unit of code (unit). So I think it's perfectly fine to feed some pre-processed code to the tests to check if the rule is working fine, and then create separate unit tests to verify that processor is working correctly as well. I think adding processor to the ruleTester will encourage bad practice.

[kaicataldo]

For the sake of discussion, can you elaborate on what bad practices this might encourage?

[JamesHenry]

I would also be curious about the bad practices you are referring to and in general I'm not sure I follow this logic - integration tests are far more valuable than unit tests, you could definitely have processor unit tests and rule unit tests individually passing but not have a working solution for your users.

Additionally, you could argue that RuleTester is already not a true unit test for rules, given they operate on ASTs, not on source code which is what you provide to RuleTester.

[ilyavolodin]

@kaicataldo I'm talking about bad practice of increasing the scope of a "unit".
@JamesHenry I don't want to argue about which types of tests are more valuable. They are supposed to test different things, and each one is valuable in their own right. You are correct about RuleTester not operating on AST, which makes it none pure unit testing library. But in my opinion it's a convenience part of the library. It would be very inconvenient to test directly on AST, so library is providing convenience feature to make it easier. But the fact of the matter is that the library is called "RuleTester", and processors are not related to rules at all. It's also quite easy to test rules and processors separately. It's also reasonable easy to create integration tests outside of RuleTester to test the whole flow.

[mysticatea]

As I have written vue/comment-directive rule in the motivation section, I have a rule that requires a specific processor. That rule is to implement <!-- eslint-disable --> directive comments in HTML templates of Vue.js. That rule makes messages at every directive comment and vue.postprocess() function filters those messages and the messages which are disabled by the directive comments. That rule requires processor option for their unittest.

[platinumazure]

@mysticatea I don't want to derail this RFC, but it almost seems to me that what you need in your case is ESLint to support AST extensions to mark certain tokens as disable comments, which ESLint would then handle as if they were disable comments in JavaScript. Then you would be able to decorate your comment tokens in the Vue parser and ESLint would just apply its own disable logic the way it normally would. But maybe I'm missing something.

[btmills]

@mysticatea if you have a moment to take a look at this: I'll be updating this RFC soon so we can use it to solve eslint/eslint#14800. Before I do, were there any additional changes you had in mind for this that I should incorporate?

[btmills]

Since the original commit, 4af9338, was written under the old CLA, it is already covered. The CLA bot will report an error, but we can ignore it for that commit because all commits are covered by a CLA.

This is ready for review.

[mdjermanovic]

!processor.supportsAutofix value will be passed as filenameOrOptions.disableFixes?

[nzakas]

I am struggling with this point. Fundamentally, it seems wrong to put this functionality into RuleTester, as it does seem beyond the scope of what testing a rule should do. Rules only run on JS code, so I’m not sure it makes sense to be able to pass non-JS code into a test.

I’m wondering if maybe we should provide a separate utility that applies processors to any text?

Maybe it’s worth taking a step back and thinking through what it is, exactly, we are trying to achieve here. Do we need a ProcessorTester?

[btmills]

For reference, we picked this RFC back up as a solution to eslint/eslint#14800, and the meeting notes for that discussion are at https://github.com/eslint/tsc-meetings/blob/main/notes/2021/2021-07-15.md#add-physicalfilename-option-into-rule-tester.

Stepping back a bit first, are we still in agreement that we ought to provide a way to test rules that care about physicalFilename?

From an implementation perspective, RuleTester uses the Linter API, so the approach in this RFC takes advantage of existing API. If we were to allow setting physicalFilename from RuleTester directly, we'd need to add additional API surface to Linter that is only used by tests, which feels icky.

Making RuleTester aware of processors doesn't bother me too much because the focus is still on testing a rule, and the physicalFilename that the rule has access to is processor-related context. If we had an easy way to test rules with fully accurate context that didn't require running the processor (and perhaps even Linter), that might be preferable. Maybe that would look like replicating a minimal bit of parsing and traversal logic inside RuleTester, which would call rules directly? Then your idea of a utility to run a processor on arbitrary text could slot in nicely.

Can you elaborate more on what a ProcessorTester might do? It sounds like it might be useful for processors, though potentially without solving the physicalFilename in rules use case this RFC addresses.

[nzakas]

I don’t have good answers for your questions right now, I’m just concerned that we are going down a path that has led to pain the past, which is when we start adding more and more responsibility to an object to the point where it becomes unwieldy (see Linter) rather than taking a step back and thinking about how we might rethink the solution given what we know now.

So yes, the easiest solution is to add processor support into RuleTester — I just think that maybe there’s a more fundamental issue, like perhaps we need to rethink how Linter deals with physical file names to enable better testing.

(I’ll think more on this when I’m less tired.)

[btmills]

This probably isn’t feasible, but throwing it out anyway: as you pointed out, RuleTester already does a lot internally, some of which parallels logic elsewhere in the engine. But its API surface is relatively small, which gives us a lot of freedom. What if we refactored RuleTester to use the ESLint API instead of Linter, then allow consumers to replace the default by passing in a pre-configured ESLint instance to the constructor? That second step might only work with flat config so that it doesn’t depend on modules being present in the file system. Then, if this works at all, if someone wanted to test a rule in a processor context, they could configure ESLint with a processor, then use the test case’s existing name option to trigger the processor.

[nzakas]

I’m not sure that changes the equation here. My concern is that I’m not sure RuleTester should know about processors at all; changing from Linter to ESLint just changes how processors might be implemented.

If the main concern is access to context.getPhysicalFilename(), it just seems like overkill to load a processor for such a narrow use case.

[btmills]

I’m not sure RuleTester should know about processors at all

Ah, now I understand your objection. Thank you. In theory, I'm inclined to agree. But we may have already crossed this bridge, so let's see if this counterargument is compelling:

Rules themselves already know about processors in the form of getFilename() vs getPhysicalFilename(). Therefore shouldn't RuleTester also know about processors?

[nzakas]

I think the difference is that rules know about processor meta data, but not about processors themselves. In theory, the filename and physical filename could be populated by something other than a processor, and I’m more inclined to feel like that’s a better direction than running an actual processor to fill in those values.