Lexon v1.0.0-rc.1 — A Full-stack Language for LLM Systems

Every time I tried to build a serious LLM workflow, I ended up juggling scripts, notebooks, glue services, and half a dozen “context” hacks. I wanted a language where async orchestration, validation, RAG, agents, and now structured memory are first-class—not bolted on after the fact. Lexon is my answer: an LLM-first programming language with a deterministic runtime, strong governance, and batteries included. This RC introduces the biggest addition so far, Structured Project Memory, but it sits next to peers like MCP, sessions, merge/fallback/ensemble, arbitrage, multioutput, and advanced RAG. Here’s the tour.

0. Copy-paste quickstart

Run one command, watch async orchestration + merge happen, and be done before the kettle boils:

cargo run -q -p lexc-cli -- compile --run samples/01-async-parallel.lx
pub fn main() {
  set_default_model("simulated");

  let [outline, slogans] = ask_parallel([
    ask { user: "Outline the release checklist for Lexon RC.1"; temperature: 0.1; },
    ask { user: "Give me two motivating one-liners for the launch"; temperature: 0.4; }
  ]);

  let merged = ask_merge(outline, slogans, "Return two concise bullet points for kickoff");
  print(merged);
}

No shell scripts, no YAML pipelines: lexc-cli compiles and runs the IR, and the deterministic runtime simulates the LLMs until you provide real API keys.

Toolchain assumptions

1. Lexon in three minutes

If you’re tired of stitching together Python notebooks, LangChain pipelines, or orchestration DAGs just to run prompts with context and validation, Lexon is the opposite experience: a real language with LLM-first primitives, governance, and structured memory built in. Instead of spinning up extra frameworks and services for every new feature, Lexon gives you:

That lexon.toml (shipped in v1.0.0-rc.1/lexon.toml) is where you declare [system] default_provider, [providers.<name>] blocks, web_search presets, sandbox flags, and structured-memory backends. No extra bootstrap layer required.

Hello world stays simple:

pub fn main() {
  set_default_model("simulated");
  let message = ask("Say hello from Lexon");
  print(message);
}

Execution goes through lexc-cli. Offline simulations are the default; setting OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, or custom provider blocks in lexon.toml seamlessly switches to real models.

How it compares

2. LLM orchestration suite (beyond any single feature)

The orchestration surface is intentionally broad because real apps need more than a lone ask():

Example pipeline:

let ds = load_csv("samples/triage/tickets.csv");
let urgent = filter(ds, 'priority == "high"');
save_json(urgent, "output/high_tickets.json");

let brief = ask_safe {
  user: "Summarize the high priority tickets",
  validation: "basic",
  max_attempts: 2
};

print(brief);

Dual-role prompt (system + user) with guardrails:

let summary = ask {
  system: "You are Lexon's semantic memory layer. Be precise.";
  user: "Summarize the runtime guide in two bullet points.";
  model: "openai:gpt-4o-mini";
  temperature: 0.2;
  max_tokens: 256;
};

Hybrid search + fusion + answer:

let hits = memory_index.hybrid_search("before_action hook", 5);
let context = rag.fuse_passages(hits, 3);
let answer = ask {
  system: "Use only the provided context.";
  user: strings.join([
    "Context:\n", context, "\n\nQuestion: How do before_action hooks enrich agents?"
  ], "")
};
print(answer);

3. MCP agents, tooling, and observability

Agents aren’t useful if you can’t govern them, cancel them, or see what they’re doing. MCP support comes built-in—you can launch stdio or WebSocket MCP servers directly from Lexon, register tools with quotas, and stream progress/cancelation signals without extra glue:

Everything funnels through the same governance rails: structured memory, RAG queries, HTTP calls, MCP tools, and multioutput share telemetry, budgets, and deterministic behavior.

Start servers and hook context:

# stdio server
cargo run -q -p lexc -- --mcp-stdio

# WebSocket server with custom addr
cargo run -q -p lexc -- --mcp-ws --mcp-addr 127.0.0.1:9443
before_action use_context project="lexon_demo" topic="runtime";

let supervisor = agent_create("runtime_supervisor", """{"model":"openai:gpt-4o-mini","budget_usd":0.15}""");
let report = agent_run(supervisor, "Produce the deployment checklist for RC.1", """{"deadline_ms": 30000}""");
print(report);

Telemetry in practice

Lexon’s OTEL hooks are baked into the runtime. Flip a single env var and every scheduler hop, ask call, structured-memory write, or MCP tool execution emits spans:

LEXON_OTEL=1 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  cargo run -q -p lexc-cli -- compile --run samples/01-async-parallel.lx

The bundled OTLP smoke collector (cargo make otel-smoke) surfaces spans such as lexon.scheduler.execute, lexon.ask.request, and lexon.memory.remember_raw, each annotated with model, tokens, duration, and budget metadata—proof that this stack is observability-ready.

4. Structured Project Memory (spotlight)

This RC’s headline feature is a native “second brain” for each Lexon project. It doesn’t replace RAG; it sits above it, curating the knowledge humans actually care about before any retrieval call.

4.1 Architecture

  1. Semantic layer (LLM-assisted or manual) creates canonical MemoryObjects with path, kind, raw, summary_micro/short/long, tags, metadata, relevance, pinned, timestamps, and policies.
  2. Tree/index backends (pluggable):

4.2 Language primitives

let _ = memory_space.create("lexon_demo", """{"reset": true}""");

let obj = remember_raw(
  "lexon_demo",
  "decision",
  read_file("docs/runtime_decision.md"),
  """{"project": "runtime", "path_hint": "lexon/runtime/decisions"}"""
);

let bundle = recall_context(
  "lexon_demo",
  "runtime",
  """{"limit": 3, "include_raw": true, "freeze_clock": "2025-01-01T00:00:00Z"}"""
);

print(obj);
print(bundle);

remember_raw talks to the configured provider (OpenAI, Anthropic, Google, Ollama, HF, custom) to infer structure, tags, relevance, pinning suggestions, and metadata; remember_structured ingests pre-built payloads. Both obey budgets, retries, telemetry, and deterministic testing features such as freeze_clock.

4.3 Why it matters

5. Where it fits in the broader stack

Structured memory is a peer to RAG, MCP, sessions, merge/fallback/ensemble, arbitrage, multioutput, and other orchestration features—one more first-class capability, not the only story.

6. Lexon feature catalog (expanded)

If you just need the checklist, here it is:

7. Stability map (RC vs 1.1)

GA exit criteria: p95 runtime <1.2× baseline (samples/apps/research_analyst), <1% token-budget regression on samples/memory/structured_semantic, and OTEL spans present for every tool/ask call in CI smoke tests.

8. Samples & how to actually use Lexon

These are the programs I run to prove Lexon still “gets it right” end-to-end:

Commands:

cargo build --workspace
cargo run -q -p lexc-cli -- compile --run samples/00-hello-lexon.lx
cargo make samples-smoke
cargo make samples-snapshot

Switch structured memory backend:

LEXON_MEMORY_BACKEND=hybrid cargo run --bin lexc -- samples/memory/structured_semantic.lx

Use real providers by exporting API keys and editing lexon.toml (default_provider, per-provider defaults). Without keys, everything is deterministic and runs offline.

9. Roadmap snapshot

We gate GA on three metrics: median cargo make samples-smoke, p95 runtime for samples/apps/research_analyst, and structured-memory recall accuracy on the golden sample. When those stay inside budget for two consecutive runs, RC graduates to 1.0.

10. Getting started

  1. Clone github.com/lexon-lang/lexon (RC pinned to Rust 1.82).
  2. cargo build --workspace.
  3. Run samples in simulated mode or export provider keys for real runs.
  4. Play with samples/memory/structured_semantic.lx using different backends (basic, patricia, raptor, hybrid).
  5. Read README.md, DOCUMENTATION.md, communication/lexon_memory_features.md for the deep details.

Lexon already drives MCP agents, ETL pipelines, copilots, RAG flows, and now high-signal project memory—all from one language. If you try it, let me know what you build; I’m still the only person maintaining this thing, and your feedback shapes the backlog.

Repository: github.com/lexon-lang/lexon
Docs: README.md, DOCUMENTATION.md, communication/lexon_memory_features.md
Contact: open an issue/PR or share your demo referencing Lexon + Structured Project Memory.

11. What to try next