Skip to main content

Build Local AI Apps with the VeloxQuant Node.js SDK

· 3 min read
Rajveer Rathod
Rajveer Rathod
Author of VeloxQuant-MLX

Local AI development should feel familiar to web developers: install a package, choose a model, and start building. The VeloxQuant Node.js SDK makes it easier to create private, hardware-aware AI applications with JavaScript and TypeScript on Apple Silicon.

Install the SDK​

Install the published package from npm:

npm install @veloxquant/sdk

You can find the package, release history, and usage information here:

View @veloxquant/sdk on npm

The SDK is designed for Apple Silicon Macs and works with local models supported by VeloxQuant-MLX.

What you can build today​

The SDK brings hardware-aware local inference and KV-cache optimization into familiar JavaScript APIs.

Hardware-aware recommendations​

Get a recommendation based on the model size, context length, device, and available memory:

import { VeloxQuant } from "@veloxquant/sdk";

const vq = new VeloxQuant();

const recommendation = await vq.recommendModel({
  modelClass: "7B",
  goal: "max_context",
  seqLen: 32768,
});

console.log(recommendation.recommendation.method);
console.log(recommendation.recommendation.rationale);

Memory estimation​

Estimate KV-cache requirements before loading a model:

const estimate = await vq.memory.estimate({
  seqLen: 32768,
  headDim: 128,
  nLayers: 32,
});

console.log(estimate.recommendedMethod);
console.log(estimate.memorySavedBytes);

Local model serving​

Start a local model and use the OpenAI-compatible chat interface:

const model = await vq.load({
  model: "mlx-community/Qwen3-4B-4bit",
  optimize: true,
});

const response = await model.chat({
  prompt: "Explain KV-cache compression in simple terms.",
  maxTokens: 200,
});

console.log(response.message.content);
await model.stop();

Streaming is also supported:

for await (const chunk of model.stream({
  prompt: "Write a short welcome message.",
})) {
  process.stdout.write(chunk.text);
}

Method discovery​

Inspect the available compression methods and choose the right family for your application:

const methods = await vq.models.list({ servableOnly: true });

for (const method of methods.methods) {
  console.log(method.name, method.family, method.serveTierLabel);
}

Benchmarks​

Measure real performance on your machine:

const result = await vq.benchmark({
  model: "mlx-community/Qwen3-4B-4bit",
});

console.log(result.tokensPerSecond);
console.log(result.timeToFirstTokenMs);
console.log(result.toMarkdown());

Framework integrations​

The SDK includes integrations for popular JavaScript AI frameworks:

Example with the Vercel AI SDK:

import { generateText } from "ai";
import { veloxquant } from "@veloxquant/sdk/ai-sdk";

const result = await generateText({
  model: veloxquant(model),
  prompt: "Give me three ideas for a local AI app.",
});

console.log(result.text);

Command-line tools​

The package also includes a CLI:

npx veloxquant doctor
npx veloxquant recommend
npx veloxquant analyze --seq-len 32768 --head-dim 128 --n-layers 32
npx veloxquant serve --model mlx-community/Qwen3-4B-4bit

Explore the documentation​

Whether you are building a desktop assistant, a private coding tool, a long-context research application, or an on-device agent, the goal is the same: make local AI development on Apple Silicon approachable from the tools you already use.