Create a synthetic test generator, and use it to create tests for WLA and WLALINK

[vhelin]

As writing big test projects is pretty hard and time consuming I think we should invest time into making a test project generator: a smallish (hopefully) program that generates projects containing assembly source files, makefiles, linkfiles and testfiles (or tests are included in source files' comments).

The source files should contain:

• lots of sections (with section appending and discarding)
• lots of labels (local and global)
• lots of calculations
• references to labels in other files
• structs and other directives
• ...

Making such a program might be more interesting than just writing a couple of test projects manually? :) Also this way we could create bytewise big tests that are really hard to do manually...

Any ideas?

[lhsazevedo]

If I got it right, this would be a fuzz test generator?

[vhelin]

I know very little about testing, never heard about fuzz tests before. :)

[lhsazevedo]

Me too :)

Some days ago I read this article Vulnerability Discovery Against Apple Safari and the part about Grammar Based Fuzzing is quite interesting.
While their goals were different, some aspects of the technique for generating valid-but-nonsense code could serve as a reference for us, even if we come up with a different approach.

[cr1901]

@vhelin Csmith is the C version of what you're asking for. I know one of the authors; I could ask him if there's an "easy" (i.e. doesn't-require-a-research-grant's-worth-of-money) way to create such a program for an assembler.

[vhelin]

Making such a project generator for WLA DX shouldn't be a problem: create n asm files that contain x sections each using the same .MEMORYMAP, then insert random labels into these sections and make references to the labels. It should be pretty simple to get started. And while it could generate asm code it's not necessary, you can put the references and calculations into .db/.dw/.dl fields... If you want asm, just pick a random opcode from the opcodes array (needs a file like decode_z80.c, but instead of decoding it'll output asm code)...

[vhelin]

But please ask! Anything helps. :)

[cr1901]

Section 2.3 of this paper gives the overall algorithm. I wonder if there's pseudocode: https://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

[cr1901]

Here's the relevant Twitter thread: https://twitter.com/cr1901/status/1387474784011816966

Short version: We should start from scratch. We should think carefully about where the bugs are likely to be and create a fuzzer that's tuned for finding those bugs.

My guess is that a fuzzer for the macro language, directives, and label arithmetic are a good place to check. Also maybe apply AFL?