Building a Lexer and Parser for a BBS Programming Language
July 31, 2026
How do you build a programming language for a BBS? You start with the same tools every language uses: a lexer to break source code into tokens, and a parser to turn tokens into a tree. This is the story of how Phosphor’s lexer and parser work, with real code from the implementation.
The Design
Phosphor is a domain-specific language for BBS configuration and behavior. It has two layers:
Declarative — describe the BBS structure: menus, boards, channels, file areas, themes. Imperative — define behavior with event handlers, expressions, and control flow.
The language needs to be readable by sysops who aren’t programmers. It needs to support hot-reload — you save a file and the BBS reconfigures itself. And it needs to be parsed fast enough that re-parsing on every file save is instant.
Step 1: The Tokenizer
The lexer breaks a .phos source string into a list of tokens. Every token has a type, a lexeme (the raw text), an optional literal value, and a position (line and column) for error reporting.
public class PhosphorLexer {
private static final Map<String, TokenType> KEYWORDS = TokenType.keywords();
private final String source;
private final List<Token> tokens = new ArrayList<>();
private int start = 0; // start of current lexeme
private int current = 0; // current position in source
private int line = 1; // 1-based line number
private int lineStart = 0; // source index of start of current line
public PhosphorLexer(String source) {
this.source = source;
}
public List<Token> tokenize() {
while (!isAtEnd()) {
start = current;
scanToken();
}
tokens.add(new Token(TokenType.EOF, "", null, line, column()));
return tokens;
}
}
The core scanning loop is a single switch statement. Whitespace is skipped. Comments (both // line comments and /* block comments */) are consumed. Single-character tokens — (, ), {, }, [, ], :, ,, ., ?, @, % — are direct matches. Multi-character tokens — ==, !=, <=, >=, ->, ++, &&, || — use a one-character lookahead.
private void scanToken() {
char c = advance();
switch (c) {
case ' ', '\r', '\t' -> {}
case '\n' -> { line++; lineStart = current; }
case '/' -> {
if (match('/')) {
while (peek() != '\n' && !isAtEnd()) advance();
} else if (match('*')) {
blockComment();
} else {
addToken(TokenType.SLASH);
}
}
case '(' -> addToken(TokenType.LPAREN);
case ')' -> addToken(TokenType.RPAREN);
case '{' -> addToken(TokenType.LBRACE);
case '}' -> addToken(TokenType.RBRACE);
case '=' -> addToken(match('=') ? TokenType.EQEQ : TokenType.EQUALS);
case '!' -> addToken(match('=') ? TokenType.BANGEQ : TokenType.BANG);
case '<' -> addToken(match('=') ? TokenType.LTEQ : TokenType.LT);
case '>' -> addToken(match('=') ? TokenType.GTEQ : TokenType.GT);
case '-' -> addToken(match('>') ? TokenType.ARROW : TokenType.MINUS);
case '"' -> stringLiteral();
case '\'' -> charLiteral();
default -> {
if (isDigit(c)) {
numberOrDurationOrTime();
} else if (isAlpha(c)) {
identifierOrKeyword();
} else {
throw error(line, column(), "unexpected character: '%s'".formatted(c));
}
}
}
}
String Interpolation
Phosphor strings support interpolation — "Hello ${user.name}". The lexer detects ${ inside a string and consumes until the matching }, tracking nesting depth. The token is marked as INTERPOLATED_STRING so the parser knows to extract and evaluate the embedded expressions.
private void stringLiteral() {
boolean hasInterpolation = false;
StringBuilder value = new StringBuilder();
while (!isAtEnd() && peek() != '"') {
if (peek() == '$' && peekNext() == '{') {
hasInterpolation = true;
advance(); // $
advance(); // {
int depth = 1;
StringBuilder interp = new StringBuilder();
while (!isAtEnd() && depth > 0) {
char ch = advance();
if (ch == '{') depth++;
else if (ch == '}') depth--;
if (depth > 0) interp.append(ch);
}
value.append("${").append(interp).append("}");
continue;
}
// Escape sequences and normal characters...
}
TokenType type = hasInterpolation ? TokenType.INTERPOLATED_STRING : TokenType.STRING;
addToken(type, lexeme, hasInterpolation ? lexeme : value.toString());
}
Numbers, Durations, and Times
Phosphor needs three numeric types: plain integers (42), floats (3.14), and durations (30m, 7d, 2h). The lexer uses lookahead to distinguish them:
private void numberOrDurationOrTime() {
while (isDigit(peek())) advance();
if (peek() == '.' && isDigit(peekNext())) {
advance();
while (isDigit(peek())) advance();
}
// Duration suffix: 30m, 7d, 2h, 500ms
if (isAlpha(peek())) {
StringBuilder suffix = new StringBuilder();
while (isAlpha(peek())) suffix.append(advance());
String s = suffix.toString();
if (s.equals("d") || s.equals("h") || s.equals("m") || s.equals("ms") || s.equals("s")) {
addToken(TokenType.DURATION, source.substring(start, current),
parseDuration(source.substring(start, current)));
return;
}
}
// Time: 14:32:01
if (peek() == ':' && isDigit(peekNext())) {
advance();
while (isDigit(peek())) advance();
if (peek() == ':' && isDigit(peekNext())) {
advance();
while (isDigit(peek())) advance();
}
addToken(TokenType.TIME, source.substring(start, current));
return;
}
// Plain number
String num = source.substring(start, current);
if (num.contains(".")) {
addToken(TokenType.FLOAT, num, Double.parseDouble(num));
} else {
addToken(TokenType.INT, num, Long.parseLong(num));
}
}
Keywords
Keywords are stored in a TokenType.keywords() map that maps strings to token types. When the lexer encounters an identifier, it checks if it matches a keyword:
private void identifierOrKeyword() {
while (isAlphaNumeric(peek())) advance();
String text = source.substring(start, current);
TokenType type = KEYWORDS.get(text);
if (type == null) type = TokenType.IDENTIFIER;
addToken(type);
}
The keyword list includes multi-word keywords like file_area and chat_message — the lexer reads the full identifier and looks it up in the map. No special handling needed.
Step 2: The Parser
The parser is a recursive descent parser. It walks the token list, building an AST (Abstract Syntax Tree) of AstNode records. The AST is what the runtime walks to sync declarations to the database and execute handlers.
public class PhosphorParser {
private final List<Token> tokens;
private int current = 0;
public PhosphorParser(String source) {
this.tokens = new PhosphorLexer(source).tokenize();
}
public PhosphorProgram parse() {
List<AstNode> declarations = new ArrayList<>();
while (!isAtEnd()) {
declarations.add(topLevelDeclaration());
}
return new PhosphorProgram(declarations);
}
}
Top-Level Declarations
The parser dispatches on the current token type. Each declaration type has its own method:
private AstNode topLevelDeclaration() {
if (match(TokenType.BBS)) return bbsBlock();
if (match(TokenType.GROUP)) return groupDecl();
if (match(TokenType.MENU)) return menuDecl();
if (match(TokenType.BOARD)) return boardDecl();
if (match(TokenType.CHAT)) return chatDecl();
if (match(TokenType.FILE_AREA)) return fileAreaDecl(null);
if (match(TokenType.SCREEN)) return screenDecl();
if (match(TokenType.DOOR)) return doorDecl();
if (match(TokenType.API)) return apiRouteDecl();
if (match(TokenType.WEBSOCKET)) return websocketDecl();
if (match(TokenType.SYNCABLE)) return syncableDecl();
if (match(TokenType.ON)) return onHandlerDecl();
if (match(TokenType.TEST)) return testDecl(false);
// ...
throw error(peek(), "expected a top-level declaration");
}
The match method checks the current token type and advances if it matches. This is the standard recursive descent pattern — check, consume, parse.
Parsing a Board Declaration
Here’s how a board block parses. The pattern is the same for every declaration: consume the keyword, read the name, expect an opening brace, parse the body, expect a closing brace.
board "General" {
description = "General discussion"
read = ["users"]
write = ["users"]
federated = true
}
The parser reads the name as a string token, then iterates through key-value pairs inside the block. Each key is an identifier, followed by =, followed by an expression. The expression can be a string, a number, a boolean, a list, or a map.
The Match/Check/Consume Pattern
Three helper methods drive the parser:
private boolean match(TokenType type) {
if (check(type)) {
advance();
return true;
}
return false;
}
private boolean check(TokenType type) {
return !isAtEnd() && peek().type() == type;
}
private Token consume(TokenType type, String message) {
if (check(type)) return advance();
throw error(peek(), message);
}
match tries to consume a token, returning whether it succeeded. check looks ahead without consuming. consume consumes a token or throws a parse error. Every error includes the token’s line and column, so the editor can show the exact location of syntax errors.
AST Nodes as Records
Every AST node is a Java record — immutable, transparent, and compact:
public record WebSocketDecl(
String path,
boolean requireAuth,
Optional<String> requireGroup,
List<OnHandlerDecl> handlers,
List<Stmt.VarAssignment> bindings,
int line,
int column
) implements AstNode {}
Records make the AST self-documenting. You can see exactly what data each node carries. No getters, no builders, no mutable state. The parser produces records, the evaluator consumes them, and the database sync layer walks them.
Error Reporting
Parse errors include line and column so the built-in editor can highlight them:
private ParseError error(Token token, String message) {
String where = token.type() == TokenType.EOF
? " at end"
: " at '" + token.lexeme() + "'";
throw new ParseError("[line " + token.line() + ":" + token.column() + "] " + message + where);
}
The editor screen uses these to show a cursor on the exact character that caused the error. No stack traces, no Java exceptions — just “expected ‘}’ at line 42, column 15.”
The Test Strategy
Every token type, every parsing rule, every error condition gets a test first. The lexer tests cover every keyword, every operator, every escape sequence, every edge case (empty strings, unterminated comments, numbers at the end of input). The parser tests cover every block type, every expression form, every error message.
All tests run in mvn test alongside the existing Java tests. Same JUnit, same Surefire, same reports. 7,000+ tests across the entire project, and the Phosphor tests are just part of the suite.
Why Not a Parser Generator?
Tools like ANTLR can generate a parser from a grammar file. Phosphor uses a hand-written recursive descent parser instead. Three reasons:
-
Error messages. Generated parsers produce generic errors (“unexpected token FOO”). Hand-written parsers produce domain-specific errors (“expected ‘connect’, ‘message’, ‘disconnect’, or ‘update’ after ‘on’ in websocket block”).
-
Control. When a new feature needs special handling (like string interpolation, or duration suffixes), you just write it. No grammar file to update, no regeneration step.
-
Readability. The parser reads like a spec. When you want to know what
syncable "trade_signal" { ... }looks like in the AST, you read thesyncableDecl()method. It’s right there.
What’s Next
The lexer and parser are the foundation. Everything else — the evaluator, the reactive binding system, the WebSocket endpoints, the federation runtime — builds on the AST they produce. A clean parser makes a clean language. And a clean language makes a BBS that sysops actually want to configure.