Quixotic AI: AI inference for the JVM at native speed

Show HN: Jinfer – AI inference engine for the JVM. AI in a jar

LLMs, vision, embeddings, and text-to-speech, all inside the JVM. AI in a jar.

3 points by mukel 37 days ago qxotic.ai 0 comments

Summary not yet generated for this article.

Original Text

Hi HN, Some would call AI on the JVM quixotic. And so, Quixotic AI was born.

jinfer is an inference engine for the JVM: chat, vision, audio, embeddings, reranking, and TTS. No Python runtime, no ONNX, no Docker containers, no sidecar process, no IPC. Finally, AI in a jar.

The stack underneath is built for the JVM rather than bolted onto it:

    toknroll: pure-Java tokenizers, zero dependencies
    gguf / safetensors: read and write llama.cpp and HuggingFace model formats
    jam: quantized matmul kernels, competitive with llama.cpp on CPU
    jota: Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo

It ships with integrations for Spring AI and LangChain4j, and is compatible with GraalVM Native Image for low-overhead, self-contained binaries with millisecond startup.

This is an early release (CPU only): I'd especially like feedback on the API surface.

Original Text

Comments (0 of ) ⌛