Inferencer

Inferencer: Run and Deeply Control Local AI Models

Overview

Inferencer is a powerful platform designed for running artificial intelligence (AI) models locally. It prioritizes user control, privacy, and the flexibility of AI operations. The platform’s capabilities include inspecting token probabilities, serving models over a network, and streaming large models with memory offloading. This tool is especially useful for developers, researchers, and businesses looking to harness AI without compromising data privacy or security.

Deep Control Over AI Models

The core philosophy of Inferencer is that AI should be transparent and manageable. With a strong focus on local processing, Inferencer ensures that users have complete control over their AI models, avoiding the pitfalls of black-box algorithms. By default, all AI operations occur offline on the user's device, meaning that no data is transmitted to the cloud. This feature is particularly beneficial for organizations dealing with sensitive information or requiring stringent compliance with data protection regulations.

Features and Functionalities

Token Inspection

Inferencer provides robust tools for inspecting token probabilities. Users can access detailed insights about what the model is thinking during processing. The tool includes features such as:

• Token Entropy: Instantly view contentious tokens, helping gauge confidence levels in generated outputs.
• Token Selection: Explore alternative branches by specifically selecting tokens.
• Token Exclusion: Exclude undesired tokens from outputs to tailor responses further.

Private Server Capabilities

The platform supports inferencing over local networks or the internet with added security measures, including SSL encryption and IP protection. Inferencer is compatible with APIs such as Ollama and OpenAI, making it easier for developers to integrate with existing applications. Additional features include:

• Agent Support: Built-in tools for popular coding environments, enabling seamless integration with services like GitHub Copilot and Vibe CLI.
• Persistent Prompt Caching: Caches processed prompts to deliver rapid responses for repeated inputs.
• Distributed Inference: Allows linking multiple computers to collaboratively process larger models.

Expert Control

Inferencer is versatile in its control mechanisms, allowing users to adjust the performance of Mixture of Experts (MoE) models according to their needs. This feature enables faster inference rates or enhanced model intelligence by controlling the number of experts utilized.

Prompt Prefilling and Custom Tool Calls

Users can prefill prompts to seed initial responses from models. This capability simplifies the task of guiding the model’s responses and omitting unnecessary preambles. The tools editor feature allows users to enable built-in tools or create custom tools, enhancing the model's functional scope.

Markdown Rendering and More Features

Inferencer supports advanced markdown rendering, including LaTeX for specialized outputs. Notable additional features include:

• Model Streaming: Stream large models using custom implementations suited for low-resource devices.
• Model Compression: Implement optional memory conservation strategies during periods of inactivity.
• Batching: Enhance throughput by combining multiple requests into single processing runs.

Pricing Models

Inferencer offers two pricing tiers tailored to diverse user needs:

Free Plan

• Features Include:
  • Private on-device AI
  • Unlimited processing
  • Markdown rendering
  • Basic model control settings
  • Limited inference server and cache
• Restrictions Apply: Users on the free plan have limited access to advanced features, including specified token management and distributed computing capabilities.

Professional Plan

• Monthly Fee: $9.99 or $99.99 per year.
• Features Include:
  • All free plan features with enhancements such as encrypted inference servers, unlimited prompt caching, and unrestricted access to custom tools.
  • Enhanced performance capabilities like model streaming and distributed compute features.

This structured pricing approach ensures that users can select a plan that best suits their requirements – whether for personal experimentation or professional development.

Conclusion

Inferencer is an innovative tool for anyone working with AI models locally. Its focus on privacy, control, and educational capabilities distinguishes it from conventional cloud-based AI platforms. By enabling rapid local processing, supporting a variety of models, and offering extensive features for exploring and refining AI outputs, Inferencer is particularly beneficial for data scientists, AI developers, and businesses that prioritize data security. The availability of a free tier also makes it accessible to those who wish to experiment with local AI modeling before committing to a professional plan.