[Xlang] Fory Scheam IDL and Compiler

[chaokunyang]

Feature Request

Design and implement the Fory Definition Language (FDL) - a schema-first Interface Definition Language and compiler for Apache Fory that enables cross-language serialization with consistent type definitions, automatic code generation, and full support for Fory's advanced features like reference tracking and polymorphism.

Is your feature request related to a problem? Please describe

The Cross-Language Consistency Problem

Apache Fory excels at high-performance cross-language serialization, but currently lacks a schema-first development workflow.

The schema-less works great if we have only one language, because we can serialize domain objects direclty

But for multiple languages, this creates several significant challenges:

1. Manual Type Synchronization
Teams must manually define matching type definitions in every language they use. For a simple Person type used across Java, Python, Go, Rust, and C++, developers must:

• Write 5 separate class/struct definitions
• Ensure field names, types, and orders match exactly
• Manually register each type with consistent IDs
• Keep all definitions in sync as the schema evolves

2. Error-Prone Type Registration
Fory's cross-language serialization requires consistent type IDs across all languages. Manual registration is error-prone:

// Java
fory.register(Person.class, 100);

# Python - must use exact same ID!
fory.register(Person, type_id=100)

A single mismatched ID causes deserialization failures that are hard to debug.

3. Reference Tracking Inconsistency
Fory's powerful reference tracking (ref modifier) must be consistently applied across languages. Manual configuration often leads to:

• Memory issues when one language tracks references but another doesn't
• Circular reference errors in some languages but not others
• Inconsistent object identity semantics

4. Migration Friction from Protocol Buffers
Many teams want to migrate from Protocol Buffers to Fory for:

• Better performance
• Native reference tracking and polymorphism support
• Simpler cross-language object passing without conversion

However, protobuf-generated classes contain internal state (memoized fields, weak references) that makes them unsuitable for direct Fory serialization. Users must manually redefine all types as plain POJOs.

5. No Single Source of Truth
Without a schema file, there's no authoritative definition of the data model. This leads to:

• Documentation drift
• Inconsistent field naming conventions across languages
• Difficulty onboarding new team members
• No automated validation of cross-language compatibility

Describe the solution you'd like

Build a complete FDL (Fory Definition Language) ecosystem that provides schema-first development for Fory with the following components:

Phase 1: Core Language & Compiler (PR #3106) ✅

1.1 FDL Language Specification

File Structure:

// Package declaration
package mycompany.models;

// Import other FDL files
import "common/types.fdl";

// Options (protobuf-style or bracket-style)
option java_package = "com.mycompany.models";
option java_multiple_files = true;

// Enum definition with type ID
enum Status [id=100] {
    UNKNOWN = 0;
    ACTIVE = 1;
    INACTIVE = 2;
}

// Message definition with nested types
message User [id=101] {
    string name = 1;
    int32 age = 2;
    optional string email = 3;        // Nullable field
    ref Profile profile = 4;          // Reference-tracked field
    repeated string tags = 5;         // List field
    map<string, int32> scores = 6;    // Map field
    
    // Nested enum
    enum Role [id=102] {
        USER = 0;
        ADMIN = 1;
    }
    
    // Nested message
    message Profile [id=103] {
        string bio = 1;
        optional string avatar_url = 2;
    }
    
    Role role = 7;
    repeated Profile profiles = 8;
}

Supported Constructs:

• package - Namespace declaration
• import - File imports with search path support
• option - File/message/field-level configuration
• enum - Enumeration types with explicit values
• message - Structured types with fields
• optional - Nullable field modifier
• ref - Reference-tracked field modifier
• repeated - List/array field modifier
• map<K,V> - Map/dictionary field type
• reserved - Reserved field numbers/names for evolution
• [id=N] - Explicit type ID assignment

1.2 Type System

Primitive Types:

FDL Type	Java	Python	Go	Rust	C++
bool	boolean	bool	bool	bool	bool
int8	byte	int	int8	i8	int8_t
int16	short	int	int16	i16	int16_t
int32	int	int	int32	i32	int32_t
int64	long	int	int64	i64	int64_t
uint8	short	int	uint8	u8	uint8_t
uint16	int	int	uint16	u16	uint16_t
uint32	long	int	uint32	u32	uint32_t
uint64	BigInteger	int	uint64	u64	uint64_t
float32	float	float	float32	f32	float
float64	double	float	float64	f64	double
string	String	str	string	String	std::string
bytes	byte[]	bytes	[]byte	Vec<u8>	std::vector<uint8_t>

Collection Types:

FDL Type	Java	Python	Go	Rust	C++
repeated T	List<T>	list[T]	[]T	Vec<T>	std::vector<T>
map<K,V>	Map<K,V>	dict[K,V]	map[K]V	HashMap<K,V>	std::unordered_map<K,V>

1.3 Compiler Frontend

Components:

• Lexer (fory_compiler/parser/lexer.py): Hand-written tokenizer for FDL syntax
• Parser (fory_compiler/parser/parser.py): Recursive-descent parser producing AST
• AST (fory_compiler/parser/ast.py): Schema, Message, Enum, Field, Import nodes
• Validator: Comprehensive schema validation
  • Duplicate type names detection
  • Duplicate type ID detection
  • Unknown type reference detection
  • Duplicate field number detection
  • Circular import detection
• Import Resolver: Multi-path import resolution with cycle detection

1.4 Code Generators

Java Generator:

• Generates POJOs with getters/setters
• Adds @ForyField annotations for field configuration
• Generates equals(), hashCode(), toString() methods
• Generates registration helper class
• Supports Java records via use_record_for_java_message option
• Handles nested types as static inner classes

Python Generator:

• Generates @dataclass classes with type hints
• Generates registration function
• nested types defined in enclosing data class
• Maps FDL types to Python native types

Go Generator:

• Generates structs with Fory struct tags
• Generates registration function
• Nested types with ParentChild naming
• Uses appropriate Go types (int32, string, etc.)

Rust Generator:

• Generates structs with #[derive(Fory, Clone, Debug, PartialEq)]
• Generates registration function
• nested types with enclosing type name snaked_cased one as module name
• Uses Rust idioms (Option, Vec, HashMap<K,V>)

C++ Generator:

• Generates structs with FORY_STRUCT macro
• Generates FORY_FIELD_INFO for field metadata
• Generates registration helper function
• Uses C++17 types (std::optional, std::vector, etc.)

1.5 CLI & Tooling

Command Line Interface:

# Install the compiler
cd compiler && pip install -e .

# Basic usage
fory compile schema.fdl --lang java,python,go,rust,cpp -o output/

# Language-specific output directories
fory compile schema.fdl \
    --java_out=java/src/main/java \
    --python_out=python/src \
    --go_out=go/pkg \
    --rust_out=rust/src \
    --cpp_out=cpp/include

# Import paths
fory compile schema.fdl -I path/to/imports -I another/path

Phase 2: Advanced Features

2.1 Schema Evolution Syntax

• Field deprecation with deprecated option
• Field removal with reserved declarations
• Forward/backward compatibility validation

2.2 Build System Integration

• Maven plugin for Java projects
• Gradle plugin for Java/Kotlin projects
• Bazel rules for all languages
• CMake integration for C++ projects
• Cargo build script for Rust projects

2.3 IDE Support

• VS Code extension with syntax highlighting
• IntelliJ plugin with code completion
• Language server protocol (LSP) implementation

2.4 Advanced Code Generation

• Builder pattern generation

2.5 Protobuf Migration Tools

• .proto to .fdl converter
• Compatibility layer for gradual migration
• Type mapping documentation

Phase 3: Ecosystem Integration (Future PRs)

3.1 Framework Integration

• gRPC-like RPC framework with FDL service definitions

3.2 Validation & Testing

• Cross-language roundtrip test generation
• Schema compatibility checker

Describe alternatives you've considered

Alternative 1: Continue Schema-less Usage

Approach: Keep manual type definition and registration in each language.

Pros:

• No new tooling to learn or maintain
• Maximum flexibility for language-specific customization
• No build step for schema compilation

Cons:

• High error rate in cross-language projects
• Significant developer time spent on synchronization
• No single source of truth
• Difficult to validate cross-language compatibility
• Poor developer experience

Decision: Rejected for projects with more than 2 languages or complex schemas.

Alternative 2: Use Protocol Buffers with Fory Runtime

Approach: Define schemas in .proto files, generate protobuf classes, then serialize with Fory.

Pros:

• Mature, well-documented IDL
• Extensive tooling ecosystem
• Familiar to many developers

Cons:

• Protobuf-generated classes have internal state (memoized fields, weak references) unsuitable for Fory
• Cannot use Fory-specific features (reference tracking, polymorphism)
• Requires conversion layer between protobuf and Fory types
• Performance overhead from conversion
• Conceptual mismatch between protobuf's message model and Fory's object graph model

Decision: Rejected due to fundamental incompatibility with Fory's serialization model.

Additional context

Design Principles

1. Native Code Generation: Generated code should use language-native constructs (POJOs, dataclasses, structs) with no runtime overhead
2. Protobuf Familiarity: Syntax should feel familiar to protobuf users while supporting Fory-specific features
3. Single Source of Truth: One .fdl file defines the schema for all languages
4. Type Safety: Generated code should be type-safe in each target language
5. Fory Feature Parity: Support all Fory features including reference tracking and polymorphism

Implementation Status

Component	Status	PR
FDL Language Specification	✅ Complete	#3106
Lexer & Parser	✅ Complete	#3106
Schema Validation	✅ Complete	#3106
Java Code Generator	✅ Complete	#3106
Python Code Generator	✅ Complete	#3106
Go Code Generator	✅ Complete	#3106
Rust Code Generator	✅ Complete	#3106
C++ Code Generator	✅ Complete	#3106
CLI Tool	✅ Complete	#3106
Documentation	✅ Complete	#3106
Cross-language Tests	✅ Complete	#3106
Schema Evolution	✅ Complete	#3106
Proto Parser	✅ Complete	#3106
Build Plugins	🔲 Planned	-
IDE Support	🔲 Planned	-

Related Issues

• ProtoBuffer compatibility support #1197 - ProtoBuffer compatibility support (original feature request)
• [Question] For the Java PB object generated by Protobuf, the serialization and deserialization using Fury are slower than using Protobuf. #1945 - Protobuf generated objects serialization challenges
• RoadMap for 1.0 #1017 - Roadmap for 1.0
• [Xlang] Add unsigned int8/16/32/64 to xlang serialization spec #3110 - Unsigned integer type support (completed, required for FDL)

Documentation

• FDL Overview: docs/compiler/index.md
• FDL Syntax Reference: docs/compiler/fdl-syntax.md
• Type System: docs/compiler/type-system.md
• Compiler Guide: docs/compiler/compiler-guide.md
• Generated Code Reference: docs/compiler/generated-code.md
• Protobuf vs FDL Comparison: docs/compiler/proto-vs-fdl.md

Example: Complete Workflow

1. Define Schema (addressbook.fdl):

package addressbook;

message Person [id=100] {
    string name = 1;
    int32 id = 2;
    optional string email = 3;
    repeated PhoneNumber phones = 4;
    
    enum PhoneType [id=101] {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
    }
    
    message PhoneNumber [id=102] {
        string number = 1;
        PhoneType type = 2;
    }
}

message AddressBook [id=103] {
    repeated Person people = 1;
}

2. Generate Code:

fory compile addressbook.fdl --lang java,python -o generated/

3. Use Generated Code:

Java:

Fory fory = Fory.builder().withLanguage(Language.XLANG).build();
AddressbookForyRegistration.register(fory);

Person person = new Person();
person.setName("Alice");
person.setId(123);
byte[] data = fory.serialize(person);

Python:

import pyfory
from addressbook import Person, register_addressbook_types

fory = pyfory.Fory(xlang=True)
register_addressbook_types(fory)

person = Person(name="Alice", id=123)
data = fory.serialize(person)

4. Cross-Language Interoperability:
Data serialized in Java can be deserialized in Python and vice versa, with guaranteed type compatibility.