Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling streaming input over stdin #1

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Handling streaming input over stdin #1

wants to merge 2 commits into from

Conversation

tsnl
Copy link
Owner

@tsnl tsnl commented Jul 31, 2023

I like using sexpfmt to format log messages. Unfortunately, without this PR, sexpfmt will wait for EOF before spitting out formatted output. This PR is intended to improve this situation.

Parsing large input is more cache-efficient when lexing and parsing occur in separate passes. Parser and lexer generators like lex and yacc (or their modern counterparts, flex and bison) typically accept input character-at-a-time. However, more modern parser frameworks, like LALRPOP and nom (the latter used in this project at the moment) rely on string input to lex and parse in separate passes.

As a compromise, we read input using the 'matching parentheses' property of S-expressions, using spaces to delimit atoms and parentheses (as well as other bookends) to delimit lists. This way, we can read each form into a temporary buffer, parse, and then print. This branch implements this approach. It is not as elegant as the Yacc-like approach since we must effectively lex twice, but will do for now and should be functionally equivalent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant