mermaid/instructions.md

🚀 Flowchart Parser Migration: Phase 2 - Achieving 100% Test Compatibility

📊 Current Status: Excellent Foundation Established

✅ MAJOR ACHIEVEMENTS COMPLETED:

1. ✅ Comprehensive Test Suite - All 15 JISON test files converted to Lezer format
2. ✅ Complex Node ID Support - Grammar enhanced to support real-world node ID patterns
3. ✅ Core Functionality Working - 6 test files with 100% compatibility
4. ✅ Grammar Foundation - Lezer grammar successfully handles basic flowchart features

📈 CURRENT COMPATIBILITY STATUS:

✅ FULLY WORKING (100% compatibility):

• lezer-flow-text.spec.ts - 98.2% (336/342 tests) ✅
• lezer-flow-comments.spec.ts - 100% (9/9 tests) ✅
• lezer-flow-interactions.spec.ts - 100% (13/13 tests) ✅
• lezer-flow-huge.spec.ts - 100% (2/2 tests) ✅
• lezer-flow-direction.spec.ts - 100% (4/4 tests) ✅
• lezer-flow-md-string.spec.ts - 100% (2/2 tests) ✅

🔶 HIGH COMPATIBILITY:

• lezer-flow.spec.ts - 76% (19/25 tests) - Comprehensive scenarios

🔶 MODERATE COMPATIBILITY:

• lezer-flow-arrows.spec.ts - 35.7% (5/14 tests)
• lezer-flow-singlenode.spec.ts - 31.1% (46/148 tests)

🔶 LOW COMPATIBILITY:

• lezer-flow-edges.spec.ts - 13.9% (38/274 tests)
• lezer-flow-lines.spec.ts - 25% (3/12 tests)
• lezer-subgraph.spec.ts - 9.1% (2/22 tests)
• lezer-flow-node-data.spec.ts - 6.5% (2/31 tests)
• lezer-flow-style.spec.ts - 4.2% (1/24 tests)

❌ NO COMPATIBILITY:

• lezer-flow-vertice-chaining.spec.ts - 0% (0/7 tests)

🎯 MISSION: Achieve 100% Test Compatibility

Goal: All 15 test files must reach 100% compatibility with the JISON parser.

Phase 2A: Fix Partially Working Features 🔧

Target: Bring moderate compatibility files to 100%

Phase 2B: Implement Missing Features 🚧

Target: Bring low/no compatibility files to 100%

---

🔧 PHASE 2A: PARTIALLY WORKING FEATURES TO FIX

1. 🎯 Arrow Parsing Issues (lezer-flow-arrows.spec.ts - 35.7% → 100%)

❌ Current Problems:

• Double-edged arrows not parsing: A <--> B, A <==> B
• Direction parsing missing: arrows don't set proper direction
• Complex arrow patterns failing

✅ Implementation Strategy:

1. Update Grammar Rules - Add support for bidirectional arrow patterns
2. Fix Direction Logic - Implement proper direction setting from arrow types
3. Reference JISON - Check flow.jison for arrow token patterns

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-arrows.spec.ts

2. 🎯 Single Node Edge Cases (lezer-flow-singlenode.spec.ts - 31.1% → 100%)

❌ Current Problems:

• Complex node ID patterns still failing (despite major improvements)
• Keyword validation not implemented
• Special character conflicts with existing tokens

✅ Implementation Strategy:

1. Grammar Refinement - Fine-tune identifier patterns to avoid token conflicts
2. Keyword Validation - Implement error handling for reserved keywords
3. Token Precedence - Fix conflicts between special characters and operators

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-singlenode.spec.ts

3. 🎯 Comprehensive Parsing (lezer-flow.spec.ts - 76% → 100%)

❌ Current Problems:

• Multi-statement graphs with comments failing
• Accessibility features (accTitle, accDescr) not supported
• Complex edge parsing in multi-line graphs

✅ Implementation Strategy:

1. Add Missing Grammar Rules - Implement accTitle and accDescr support
2. Fix Multi-statement Parsing - Improve handling of complex graph structures
3. Edge Integration - Ensure edges work correctly in comprehensive scenarios

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-flow.spec.ts

---

🚧 PHASE 2B: MISSING FEATURES TO IMPLEMENT

1. 🚨 CRITICAL: Vertex Chaining (lezer-flow-vertice-chaining.spec.ts - 0% → 100%)

❌ Current Problems:

• & operator not implemented: A & B --> C
• Sequential chaining not working: A-->B-->C
• Multi-node patterns completely missing

✅ Implementation Strategy:

1. Add Ampersand Operator - Implement & token and grammar rules
2. Chaining Logic - Add semantic actions to expand single statements into multiple edges
3. Multi-node Processing - Handle complex patterns like A --> B & C --> D

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Parser: packages/mermaid/src/diagrams/flowchart/parser/flowParser.ts
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-vertice-chaining.spec.ts

🔍 JISON Reference:

// From flow.jison - shows & operator usage
vertices: vertex
        | vertices AMP vertex

2. 🚨 CRITICAL: Styling System (lezer-flow-style.spec.ts - 4.2% → 100%)

❌ Current Problems:

• style statements not implemented
• classDef statements not implemented
• class statements not implemented
• linkStyle statements not implemented
• Inline classes :::className not supported

✅ Implementation Strategy:

1. Add Style Grammar Rules - Implement all styling statement types
2. Style Processing Logic - Add semantic actions to handle style application
3. Class System - Implement class definition and application logic

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Parser: packages/mermaid/src/diagrams/flowchart/parser/flowParser.ts
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-style.spec.ts

🔍 JISON Reference:

// From flow.jison - shows style statement patterns
styleStatement: STYLE NODE_STRING COLON styleDefinition
classDef: CLASSDEF ALPHA COLON styleDefinition

3. 🚨 CRITICAL: Subgraph System (lezer-subgraph.spec.ts - 9.1% → 100%)

❌ Current Problems:

• Subgraph statements not parsing correctly
• Node collection within subgraphs failing
• Nested subgraphs not supported
• Various title formats not working

✅ Implementation Strategy:

1. Add Subgraph Grammar - Implement subgraph statement parsing
2. Node Collection Logic - Track which nodes belong to which subgraphs
3. Nesting Support - Handle subgraphs within subgraphs
4. Title Formats - Support quoted titles, ID notation, etc.

📁 Key Files:

• Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Parser: packages/mermaid/src/diagrams/flowchart/parser/flowParser.ts
• Test: packages/mermaid/src/diagrams/flowchart/parser/lezer-subgraph.spec.ts

4. 🔧 Edge System Improvements (lezer-flow-edges.spec.ts - 13.9% → 100%)

❌ Current Problems:

• Edge IDs not supported
• Complex double-edged arrow parsing
• Edge text in complex patterns
• Multi-statement edge parsing

✅ Implementation Strategy:

1. Edge ID Support - Add grammar rules for edge identifiers
2. Complex Arrow Patterns - Fix double-edged arrow parsing
3. Edge Text Processing - Improve text handling in edges
4. Multi-statement Support - Handle edges across multiple statements

5. 🔧 Advanced Features (Multiple files - Low priority)

❌ Current Problems:

• lezer-flow-lines.spec.ts - Link styling not implemented
• lezer-flow-node-data.spec.ts - Node data syntax @{ } not supported

✅ Implementation Strategy:

1. Link Styling - Implement linkStyle statement processing
2. Node Data - Add support for @{ } node data syntax

---

📋 IMPLEMENTATION METHODOLOGY

🎯 Recommended Approach:

Step 1: Priority Order

1. Vertex Chaining (0% → 100%) - Most critical missing feature
2. Styling System (4.2% → 100%) - Core functionality
3. Subgraph System (9.1% → 100%) - Important structural feature
4. Arrow Improvements (35.7% → 100%) - Polish existing functionality
5. Edge System (13.9% → 100%) - Advanced edge features
6. Remaining Features - Final cleanup

Step 2: For Each Feature

1. Analyze JISON Reference - Study flow.jison for grammar patterns
2. Update Lezer Grammar - Add missing grammar rules to flow.grammar
3. Regenerate Parser - Run npx lezer-generator --output flow.grammar.js flow.grammar
4. Implement Semantic Actions - Add processing logic in flowParser.ts
5. Run Tests - Execute specific test file: vitest lezer-[feature].spec.ts --run
6. Iterate - Fix failing tests one by one until 100% compatibility

Step 3: Grammar Update Process

# Navigate to parser directory
cd packages/mermaid/src/diagrams/flowchart/parser

# Update flow.grammar file with new rules
# Then regenerate the parser
npx lezer-generator --output flow.grammar.js flow.grammar

# Run specific test to check progress
cd /Users/knsv/source/git/mermaid
vitest packages/mermaid/src/diagrams/flowchart/parser/lezer-[feature].spec.ts --run

---

🔍 KEY TECHNICAL REFERENCES

📁 Critical Files:

• JISON Reference: packages/mermaid/src/diagrams/flowchart/parser/flow.jison
• Lezer Grammar: packages/mermaid/src/diagrams/flowchart/parser/flow.grammar
• Parser Implementation: packages/mermaid/src/diagrams/flowchart/parser/flowParser.ts
• FlowDB Interface: packages/mermaid/src/diagrams/flowchart/flowDb.js

🧪 Test Files (All Created):

packages/mermaid/src/diagrams/flowchart/parser/
├── lezer-flow-text.spec.ts ✅ (98.2% working)
├── lezer-flow-comments.spec.ts ✅ (100% working)
├── lezer-flow-interactions.spec.ts ✅ (100% working)
├── lezer-flow-huge.spec.ts ✅ (100% working)
├── lezer-flow-direction.spec.ts ✅ (100% working)
├── lezer-flow-md-string.spec.ts ✅ (100% working)
├── lezer-flow.spec.ts 🔶 (76% working)
├── lezer-flow-arrows.spec.ts 🔶 (35.7% working)
├── lezer-flow-singlenode.spec.ts 🔶 (31.1% working)
├── lezer-flow-edges.spec.ts 🔧 (13.9% working)
├── lezer-flow-lines.spec.ts 🔧 (25% working)
├── lezer-subgraph.spec.ts 🔧 (9.1% working)
├── lezer-flow-node-data.spec.ts 🔧 (6.5% working)
├── lezer-flow-style.spec.ts 🚨 (4.2% working)
└── lezer-flow-vertice-chaining.spec.ts 🚨 (0% working)

🎯 Success Metrics:

• Target: All 15 test files at 100% compatibility
• Current: 6 files at 100%, 9 files need improvement
• Estimated: ~1,000+ individual test cases to make pass

---

💡 CRITICAL SUCCESS FACTORS

🔑 Key Principles:

1. 100% Compatibility Required - User expects all tests to pass, not partial compatibility
2. JISON is the Authority - Always reference flow.jison for correct implementation patterns
3. Systematic Approach - Fix one feature at a time, achieve 100% before moving to next
4. Grammar First - Most issues are grammar-related, fix grammar before semantic actions

⚠️ Common Pitfalls to Avoid:

1. Don't Skip Grammar Updates - Missing grammar rules cause parsing failures
2. Don't Forget Regeneration - Always regenerate parser after grammar changes
3. Don't Ignore JISON Patterns - JISON shows exactly how features should work
4. Don't Accept Partial Solutions - 95% compatibility is not sufficient

🚀 Quick Start for New Agent:

# 1. Check current status
cd /Users/knsv/source/git/mermaid
vitest packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-vertice-chaining.spec.ts --run

# 2. Study JISON reference
cat packages/mermaid/src/diagrams/flowchart/parser/flow.jison | grep -A5 -B5 "AMP\|vertices"

# 3. Update grammar
cd packages/mermaid/src/diagrams/flowchart/parser
# Edit flow.grammar to add missing rules
npx lezer-generator --output flow.grammar.js flow.grammar

# 4. Test and iterate
cd /Users/knsv/source/git/mermaid
vitest packages/mermaid/src/diagrams/flowchart/parser/lezer-flow-vertice-chaining.spec.ts --run

---

📚 APPENDIX: JISON GRAMMAR PATTERNS

Vertex Chaining (Priority #1):

// From flow.jison - Critical patterns to implement
vertices: vertex
        | vertices AMP vertex

vertex: NODE_STRING
      | NODE_STRING SPACE NODE_STRING

Style Statements (Priority #2):

// From flow.jison - Style system patterns
styleStatement: STYLE NODE_STRING COLON styleDefinition
classDef: CLASSDEF ALPHA COLON styleDefinition
classStatement: CLASS NODE_STRING ALPHA

Subgraph System (Priority #3):

// From flow.jison - Subgraph patterns
subgraph: SUBGRAPH NODE_STRING
        | SUBGRAPH NODE_STRING BRACKET_START NODE_STRING BRACKET_END

---

Instructions for Mermaid Development

This document contains important guidelines and standards for working on the Mermaid project.

General Guidelines

• Follow the existing code style and patterns
• Write comprehensive tests for new features
• Update documentation when adding new functionality
• Ensure backward compatibility unless explicitly breaking changes are needed

Testing

• Use vitest for testing (not jest)
• Run tests from the project root directory
• Use unique test IDs with format of 3 letters and 3 digits (like ABC123) for easy individual test execution
• When creating multiple test files with similar functionality, extract shared code into common utilities

Package Management

• This project uses pnpm for package management
• Always use pnpm install to add modules
• Never use npm in this project

Debugging

• Use logger instead of console for logging in the codebase
• Prefix debug logs with 'UIO' for easier identification when testing and reviewing console output

Refactoring

• Always read and follow the complete refactoring instructions in .instructions/refactoring.md
• Follow the methodology, standards, testing requirements, and backward compatibility guidelines

Diagram Development

• Documentation for diagram types is located in packages/mermaid/src/docs/
• Add links to the sidenav when adding new diagram documentation
• Use classDiagram.spec.js as a reference for writing diagram test files

Run the tests using: vitest run packages/mermaid/src/diagrams/flowchart/parser/lezer-*.spec.ts