Iterate on prompts without code deployments. Our new Prompt Management system brings powerful composability, version control, and instant deployment to your LLM workflows.

Features

• Powerful Composability: Use typed variables anywhere - system prompts, messages, even tool schemas
• Version Control: Track, compare, and rollback prompt versions without code changes
• Instant Deployment: Reference prompts by ID through our AI Gateway - no rebuilds required
• Real-time Testing: Experiment with different models and parameters in the Playground
• Dynamic Schemas: Variables work within JSON schemas for tools and response formatting
• TypeScript Support: Full type safety with our helper types

How It Works

Save prompts in our Playground with variables like {{hc:customer_name:string}}. Test them with different inputs and models. When ready, reference the prompt ID in your API calls through the AI Gateway:

const response = await openai.chat.completions.create({
  model: "openai/gpt-4o-mini",
  prompt_id: "abc123",
  inputs: {
    customer_name: "John Doe",
    product: "AI Gateway"
  }
});

The AI Gateway compiles your saved prompt with runtime inputs and sends it to your chosen model. Update prompts in the dashboard and changes take effect immediately - no code changes or deployments needed.

Get Started

Visit the Prompts page in your dashboard to create your first prompt, or check out our documentation for detailed examples and best practices.

We’ve improved how timestamps are displayed throughout the Helicone dashboard to provide a better user experience for teams working across different timezones.

What’s New

• Automatic Timezone Detection: Requests now display in your local timezone automatically
• Clear Timezone Indicators: Timestamps now clearly show the timezone being used
• Consistent Formatting: Improved date/time formatting across all views

This update makes it easier for distributed teams to collaborate and analyze their LLM usage patterns without timezone confusion.

Analyze your LLM usage patterns by geographic location with our new country-based filtering feature.

Features

• Country Code Filtering: Filter requests by country of origin
• Geographic Analytics: Understand where your API calls are coming from
• Global Usage Patterns: See how your application is being used around the world

How It Works

The country code is determined by the Cloudflare edge server that receives your request. Since Helicone runs on Cloudflare’s global network, we can identify which region processed each request, giving you insights into where your users are located.

How to Use

In the requests dashboard, you can now:

1. Use the new country filter in the advanced filters section
2. Add country_code:US (or any ISO country code) to your search queries
3. View geographic distribution in the analytics dashboard

This helps you understand your global usage patterns and user distribution. Country data is automatically collected for all new requests going forward.

We’ve added pricing support for OpenAI’s new GPT-4o Search model, which includes built-in web search capabilities.

What’s New

• Model: gpt-4o-search
• Automatic Detection: Helicone automatically detects and applies correct pricing when you use this model
• Cost Tracking: View search-enabled model costs separately in your dashboard

The GPT-4o Search model combines GPT-4o’s capabilities with real-time web search. With this update, you can accurately track costs for search-enhanced conversations and compare them with standard GPT-4o usage.

We’ve updated pricing for all Mistral models to reflect their current rates and added support for new model variants.

What’s Updated

• Chat Models: mistral-tiny, mistral-small, mistral-medium, mistral-large (including -latest variants)
• Instruction Models: mistral-7b-instruct, mixtral-8x7b-instruct, mixtral-8x22b-instruct
• Embedding Model: mistral-embed
• Historical Data: Previous costs remain unchanged for accurate reporting

This ensures your cost tracking stays accurate as Mistral’s model offerings and pricing evolve.

Cost tracking is now available for xAI’s Grok 4 model.

What’s New

• Models: grok-4 and x-ai/grok-4 (OpenRouter)
• Automatic Detection: Costs calculated automatically when using Grok models
• Dashboard Integration: View Grok 4 usage alongside other models in your cost analytics

With this update, teams using xAI’s Grok models can track their spending and compare costs across different AI providers.

We’re thrilled to announce the launch of Helicone AI Gateway - a powerful open-source solution for routing, caching, and managing your LLM traffic at scale.

🚀 What is Helicone AI Gateway?

The AI Gateway is a high-performance proxy that sits between your application and LLM providers, offering enterprise-grade features:

• Smart Load Balancing: Distribute requests across multiple providers
• Intelligent Caching: Reduce costs with semantic caching
• Automatic Failover: Seamlessly switch providers during outages
• Rate Limiting: Protect against abuse and control costs
• Built-in Observability: Full integration with Helicone’s analytics

💻 Get Started

The AI Gateway is available as a separate open-source project:

GitHub Repository: github.com/helicone/ai-gateway

Quick start with Docker:

docker run -p 8080:8080 helicone/ai-gateway

📚 Learn More

• Read the documentation
• View the quickstart guide
• Explore configuration options

🔧 Key Features

• Multi-Provider Support: OpenAI, Anthropic, Azure, and more
• Request Routing: Route by model, cost, or custom rules
• Security: API key management and request validation
• Performance: Built in Rust for minimal latency overhead

Start using the AI Gateway today to take control of your LLM infrastructure!

We’re thrilled to announce that we now have a Go SDK for Helicone’s Helpers Package. You can now integrate any custom LLM with Helicone using the Go Manual Logger. Install it now with:

go get github.com/helicone/go-helicone-helpers

Refer to our docs on how to use it.

Introducing Smarter Sessions Design

We’ve introduced a new design for Sessions, now with session-level metrics, smarter filters, and a cleaner UI.

The new Sessions include average latency, total cost, and improved time filtering, making it easier to debug multi-step LLM workflows. The UI has also been refined for a cleaner and more intuitive experience.

Try it out in Sessions, we’d love to hear your feedback!

Refer to our docs on how to set up Sessions to start tracing your LLM workflows.

Introducing Helicone Self-Hosting

We’re excited to re-introduce Helicone self-hosting! Now you can deploy our powerful observability platform directly within your own infrastructure with a single Docker command.

This update addresses the needs of organizations with strict security and compliance requirements, allowing you to leverage Helicone’s insights into your LLM usage without sending sensitive data to third-party servers.

Why self-host?

• Complete Data Sovereignty: Keep all LLM interactions and analytics within your security perimeter.
• Protection of Intellectual Property: Ensure proprietary prompts and model data never leave your infrastructure.
• Customized Integration: Seamlessly integrate with your internal systems and security protocols.
• Cost Efficiency at Scale: Favorable economics for high-volume LLM request processing.

Get started with a single Docker command:

git clone https://github.com/Helicone/helicone.git

# See your Helicone dashboard in localhost:3000!
cd docker
docker compose up -d

Read more about our self-hosting offering in our blog.

Added support for OpenAI's new Responses API

Helicone now supports OpenAI’s new Responses API, allowing you to monitor and analyze your model’s responses seamlessly.

What is the OpenAI Responses API?

The OpenAI Responses API enables you to provide text or image inputs to generate text or JSON outputs by calling your own custom code or using built-in tools like web search or file search.

Integrating with Helicone

1. Create a Helicone account and API key.
2. Set environment variables
3. Install the OpenAI SDK:

   npm install openai
   

4. Configure your OpenAI client to use the Helicone proxy:

   import OpenAI from "openai";
   
   const openai = new OpenAI({
     apiKey: process.env.OPENAI_API_KEY,
     baseURL: "https://oai.helicone.ai/v1", // Route requests through Helicone
     defaultHeaders: {
       "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`, // Authenticate with Helicone
     },
   });
   

With this setup, any calls to the OpenAI Responses API will be automatically logged and monitored by Helicone.

For more examples, including file search, streaming, function calling, and reasoning, please refer to our documentation.

Introducing Open WebUI Integration

Today, you can start monitoring your local LLM interactions with Open WebUI in Helicone!

With the integration, you can:

• Monitor interactions across Ollama, OpenAI-compatible APIs, and custom LLM setups.
• Get a consolidated view across model types.
• Visualize and replay requests to see prompts and outputs for evaluation.
• Track local LLM performance, including response times and throughput.
• Analyze usage patterns by model in your Open WebUI setup.

We’ve just published a comprehensive guide on how to integrate Helicone with Open WebUI (formerly Ollama WebUI). This will show you how to gain comprehensive observability across all your LLM interactions, whether you’re using local Ollama models or cloud LLM APIs.

Get started: Open WebUI x Helicone docs.

Helicone is the first observability platform to support OpenAI’s Realtime API, the 4.1 model family, and more.

What’s New?

• Added cost support for gpt-4.1, gpt-4.1-mini, and gpt-4.1-nano, including detailed token pricing (prompt, completion, and cache read).
• Redesigned the Request Drawer for better viewing of requests, responses, and key metrics.
• Added support for gpt-4.1 model family in Prompts.
• Added support for streaming tool use with Anthropic models.
• Added stop sequences parameter support in the Parameters panel.
• Added Tools Editor with streaming support when creating new tools.
• Enabled evaluation runs for free tier users!
• Added ability to delete datasets with confirmation modals and user notifications.
• Added support for streaming tool use with Anthropic models.

Improvements

• Improved overall page load and view performance.
• Improved dataset page loading and reliability for request/response fetching.
• Improved datetime filtering with a calendar and time picker input.
• Updated backend logging for smarter and more traceable logs.
• Updated cache strategy to reduce database load during property filtering.
• Updated the API cost calculator and model comparison pages with better design and more consistent pricing information.
• Added a “Response Format Editor” component to allow users to specify JSON schema formats for responses.

Bug Fixes

• Fixed streamed requests for Claude to see the entire response.
• Fixed loading prompts from requests with tool calls.
• Fixed table responsiveness issues in blogs.
• Fixed an issue with member invites for certain organization tiers.
• Fixed an issue where member invites didn’t properly check organization tiers.

Helicone Now Supports OpenAI's Realtime API

We’re thrilled to announce that Helicone now supports logging of OpenAI’s Realtime API, enabling low-latency, multi-modal conversational experiences.

Seamless Integration with Helicone

Integrating OpenAI’s Realtime API with Helicone is as simple as ever. Following our standard one-line integration approach, you can immediately start monitoring performance, analyzing interactions, and gaining valuable insights into your real-time conversations.

How it Works

Connect to the Realtime API through Helicone using your preferred provider (OpenAI or Azure). Helicone acts as a proxy, allowing you to leverage our observability features without changing your core application logic.

Example: Connecting via WebSocket (OpenAI Provider)

// Simply swap with the following url:
const url =
  "wss://api.helicone.ai/v1/gateway/oai/realtime?model=gpt-4o-realtime-preview-2024-12-17";

const ws = new WebSocket(url, {
  headers: {
    // Your OpenAI Key
    Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    // Your Helicone Key
    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
    // Optional Helicone properties for enhanced analytics
    "Helicone-Session-Id": `session_123`,
    "Helicone-User-Id": "user_123",
  },
});

Key Benefits

• Effortless Setup: Integrate with the standard Helicone proxy URL and API key header.
• Real-time Monitoring: Track latency, token usage, and other critical metrics for your real-time sessions.
• Session Analysis: Utilize Helicone headers like Helicone-Session-Id and Helicone-User-Id to group and analyze conversations.
• Multi-modal Support: Monitor both text and audio interactions.
• Provider Flexibility: Works seamlessly with both OpenAI and Azure endpoints.

Get started today by updating your WebSocket connection URL and adding your Helicone API key header. For more details, check out the full OpenAI Realtime Integration documentation.

Filter UI Update: A New Look with Enhanced Features

A fresh new look for our filtering system brings enhanced performance and usability to your data exploration experience. The redesigned filter interface is now more intuitive and powerful than ever, making it easier to analyze your LLM usage data.

What’s new

• Unified filter component: All filter elements are now integrated into a single, cohesive component for a more streamlined experience
• Persistent filters: Your active filters now persist across different pages - from dashboard to requests, users, and sessions
• URL sharing support: Share your filtered views with team members by simply copying the URL
• Enhanced performance: The new implementation is more efficient, providing faster filter operations

Try it out

Experience the new filtering system across all main pages:

• Dashboard analytics
• Request logs
• User metrics
• Session tracking

We’re excited to hear your feedback on these improvements! Let us know what you think about the new filtering experience.

Online Evaluators New UI

We’ve completely redesigned the online evaluators creation and editing experience, making it more intuitive and powerful than ever. The new interface guides developers through the process with real-time feedback and advanced configuration options.

What’s new

• Streamlined form layout: A cleaner, more organized and intuitive interface
• Advanced configuration: More granular control over your evaluator settings
• Real-time feedback: Helpful validation messages to guide you through the setup
• Enhanced user experience: Every step of creating evaluators is now more straightforward

Getting started

1. Navigate to the Evaluators tab in Helicone
2. Click “Create New Evaluator” or edit an existing one
3. Create/edit an evaluator with the new interface with real-time feedback

We’re continuously working to make Helicone’s features more accessible and powerful. Share your thoughts on the new evaluators interface - we’d love to hear your feedback!

We just added support for Vertex AI Gemini integration with Helicone logging framework, including stream handling and logging capabilities for both Python and TypeScript SDKs.

There are two ways to integrate Helicone with Vertex AI’s Python SDK, Proxy and Manual Logger. The manual logger provides more granular control over logging the LLM’s responses.

Manual Logger Key Capabilities

1. Log both request parameters and responses with full context.
2. Handle streaming responses with proper chunk logging.
3. Built-in error tracking and logging.
4. Add custom metadata to your logs
5. Explicitly specify which model is being used for better analytics.
6. Automatic tracking of streaming response timing.

For integration guide and advanced configuration options, please refer to our Vertex AI documentation.

We’re excited to announce support for Groq streaming tokens in Helicone! This update enhances our integration with Groq’s API, providing better token tracking and usage monitoring for streaming responses.

What’s new

• Full support for tracking and monitoring Groq’s streaming token responses
• Accurate tracking of prompt, completion, and total tokens
• Support for both standard and Groq-specific usage reporting formats
• Fallback mechanisms ensure consistent token tracking even with varying response formats

Getting started

To use Groq streaming with Helicone:

1. Configure your Helicone proxy settings for Groq
2. Make streaming requests as usual through the Helicone proxy
3. Monitor your token usage in the Helicone dashboard

Helicone now integrates seamlessly with LangGraph, providing comprehensive observability for your graph-based AI agents and tools. With minimal configuration, you can get full visibility into your LangGraph workflows, from basic chains to complex multi-agent systems.

Capabilties

LangGraph has emerged as a powerful framework for building stateful, multi-agent LLM applications. Our integration gives you:

• Complete visibility into every LLM call within your graph
• Session tracking for understanding full conversation flows
• Performance metrics to identify bottlenecks in complex agent interactions
• Cost monitoring across your entire agent ecosystem

Supported models

The integration works with all major LLM providers supported by LangChain, including:

• OpenAI
• Anthropic
• And more to come!

For implementation details, check out our LangGraph integration guide.

We’re excited to introduce the HeliconeLogBuilder, a new approach to handling streaming responses with better error handling and a simplified workflow.

Introducing HeliconeLogBuilder

The new HeliconeLogBuilder class provides a simplified way to handle streaming LLM responses with improved error handling and async support. This new approach makes it easier to:

• Handle errors gracefully with the setError method
• Simplify stream handling with the toReadableStream method
• Use more flexible async/await patterns with sendLog
• Track proper error status codes

Example Usage with Next.js App Router

import { HeliconeManualLogger } from "@helicone/helpers";
import { after } from "next/server";
import Together from "together-ai";

const together = new Together();
const helicone = new HeliconeManualLogger({
  apiKey: process.env.HELICONE_API_KEY!,
});

export async function POST(request: Request) {
  const { question } = await request.json();
  const body = {
    model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    messages: [{ role: "user", content: question }],
    stream: true,
  };

  const heliconeLogBuilder = helicone.logBuilder(body, {
    "Helicone-Property-Environment": "dev",
  });

  try {
    const response = await together.chat.completions.create(body);
    return new Response(heliconeLogBuilder.toReadableStream(response));
  } catch (error) {
    heliconeLogBuilder.setError(error);
    throw error;
  } finally {
    after(async () => {
      // This will be executed after the response is sent to the client
      await heliconeLogBuilder.sendLog();
    });
  }
}

Benefits Over Previous Methods

The logBuilder approach offers several advantages over the previous streaming methods:

• Better Error Handling: Automatically captures and logs errors with the setError method
• Simplified Stream Handling: Combines stream processing and logging in one step
• More Flexible Async Patterns: Better support for async workflows with the sendLog method
• Proper Status Code Tracking: Accurately records HTTP status codes, including errors
• Improved Time-to-First-Token Metrics: More reliable tracking of streaming performance

This new approach is now the recommended way to handle streaming responses with Helicone. For more information, check out our Manual Logger with Streaming guide.

Property Page Enhancements: Average Prompt Tokens & Sorting

We’ve enhanced the Properties pages with new metrics and improved user experience:

• Average Prompt Tokens: Added average prompt tokens metrics to property tables, providing deeper insights into token usage patterns.
• Backend Sorting: Implemented server-side sorting for property data, enabling efficient sorting of large datasets by any column.
• Improved Loading Experience: Replaced generic loading animations with skeleton UI components that match the structure of the loaded content.
• Responsive Skeletons: Added custom skeleton loaders for property tables, metrics cards, and pie charts that adapt to both light and dark modes.

These enhancements make it easier to analyze property data by providing more metrics and a smoother, more responsive user interface. The skeleton loading states significantly reduce the perception of loading time and prevent jarring layout shifts when data loads.

The backend sorting implementation ensures that sorting large datasets is efficient and performant, as the sorting happens at the database level rather than in the browser.

We’ve enhanced our webhook functionality to improve integration with external systems:

• Combined Request/Response URL: Replaced separate URLs with a single request_response_url field that contains both request and response data in one object.
• Performance Optimization: Reduced S3 URL generation calls from two to one per webhook.
• Configurable Data Inclusion: Added includeData parameter to the webhook API to control whether enhanced data is included in payloads.
• Updated Documentation: Added examples showing how to work with the combined data structure.

These changes make webhook integration more efficient while maintaining backward compatibility with existing implementations.

For details, see our updated documentation.

Improved Streaming Support and Async Stream Parser

We’ve made significant improvements to our streaming functionality with two key updates:

Stream Fixes

We’ve resolved several issues with stream handling across different LLM providers, ensuring more reliable and consistent streaming experiences. These fixes address edge cases and improve compatibility with various streaming implementations, including:

• Better handling of stream interruptions and reconnections
• Improved error handling for streaming responses
• Enhanced compatibility with different LLM provider streaming formats
• Fixed timing calculations for streamed responses

New Streaming Methods

The HeliconeManualLogger class now includes enhanced methods for working with streams:

• logStream: Logs a streaming operation with full control over stream handling
• logSingleStream: Simplified method for logging a single ReadableStream
• logSingleRequest: Logs a single request with a response body

Example Usage with Together AI

import Together from "together-ai";
import { HeliconeManualLogger } from "@helicone/helpers";

// Initialize with properties
const helicone = new HeliconeManualLogger({
  apiKey: process.env.HELICONE_API_KEY!,
  loggingEndpoint: "https://api.worker.helicone.ai/oai/v1/log",
  headers: {
    "Helicone-Property-Environment": "production",
  },
});

export async function POST(request: Request) {
  const { question } = await request.json();

  const body = {
    model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    messages: [{ role: "user", content: question }],
    stream: true,
  } as Together.Chat.CompletionCreateParamsStreaming & { stream: true };

  const response = await together.chat.completions.create(body);
  const [stream1, stream2] = response.tee();
  helicone.logStream(
    body,
    async (resultRecorder) => {
      resultRecorder.attachStream(stream2.toReadableStream());
    },
    {
      "Helicone-User-Id": "123",
    }
  );

  return new Response(stream1.toReadableStream());
}

These improvements make working with streaming LLMs more reliable and efficient, especially for applications that require real-time responses.

We’ve enhanced our Python Asynchronous Logging integration with new methods to completely disable all logging to Helicone. This feature allows you to temporarily or permanently stop sending any data to our backend while maintaining your integration setup.

New Methods

• disable_logging(): Completely disables all logging by shutting down the Traceloop SDK
• enable_logging(): Re-enables logging if it was previously disabled

This is different from the existing disable_content_tracing() which only omits request and response content but still sends other metrics. The new functionality gives you complete control over your data privacy and is only available when using Helicone’s async integration mode.

# Example usage
from helicone_async import HeliconeAsyncLogger
from openai import OpenAI

logger = HeliconeAsyncLogger(api_key=HELICONE_API_KEY)
logger.init()

# Completely disable all logging
logger.disable_logging()

# Your OpenAI calls here - no data sent to Helicone

# Later, re-enable logging if needed
logger.enable_logging()

Improved sorting capabilities across the platform, particularly for cost request pages. This update (#3326) makes it easier to organize and analyze your data with more intuitive sorting controls.

New Claude 3.7 Sonnet Support: Full Cost Tracking and Integration

Immediate Support for Claude 3.7 Sonnet

We’re excited to announce full support for Anthropic’s latest Claude 3.7 Sonnet model (claude-3-7-sonnet-20250219), including comprehensive cost tracking and monitoring capabilities.

What’s New in Claude 3.7?

Claude 3.7 Sonnet represents Anthropic’s latest advancement in language AI, introducing groundbreaking hybrid reasoning capabilities. As announced by Anthropic, this model offers:

Core Capabilities

• First hybrid reasoning model combining quick responses and extended thinking modes
• Up to 128K token output limit for handling extensive conversations
• Same competitive pricing: $3 per million input tokens and$15 per million output tokens (including thinking tokens)

Enhanced Performance

• State-of-the-art results on SWE-bench Verified and TAU-bench
• Best-in-class coding and software development capabilities
• Advanced reasoning and analytical processing
• Improved instruction-following and task completion reliability

Real-world Applications

The model has demonstrated exceptional performance across:

• Complex codebase analysis and management
• Advanced tool usage and automation
• Full-stack development and web applications
• Production-ready code generation with reduced errors
• Large-scale refactoring and debugging

Cost Tracking and Integration

Our platform provides comprehensive monitoring with:

Usage Analytics

• Full cost tracking for both standard and extended thinking modes
• Precise token counting for inputs and outputs
• Real-time usage monitoring and analytics
• Updated pricing calculator with latest model rates

Integration Features

• Seamless integration with existing Anthropic workflows
• Support for both API and direct integration paths
• Comprehensive request-level analytics
• Advanced monitoring dashboards

How to Use

To start using Claude 3.7 Sonnet with Helicone:

1. Update your model parameter to claude-3-7-sonnet-20250219
2. Refer to our Anthropic Integration Guide for implementation details
3. Monitor costs and usage through your Helicone dashboard

Supported Features

• Real-time cost and usage tracking
• Extended thinking mode support and monitoring
• Comprehensive token usage analytics
• Full integration with existing workflows
• Advanced request-level insights

Learn More About Claude 3.7 Sonnet

We’re excited to launch Auto-Improve, an intelligent prompt optimization tool that helps you write more effective LLM prompts. While traditional prompt engineering requires extensive trial and error, Auto-Improve analyzes your prompts and suggests improvements instantly.

How it Works

1. Click the Auto-Improve button in the Helicone Prompt Editor
2. Our AI analyzes each sentence of your prompt to understand:

• The semantic interpretation
• Your instructional intent
• Potential areas for enhancement

3. Get a new suggested optimized version of your prompt

Key Benefits

• Semantic Analysis: Goes beyond simple text improvements by understanding the purpose behind each instruction
• Maintains Intent: Preserves your original goals while enhancing how they’re communicated
• Time Saving: Skip hours of prompt iteration and testing
• Learning Tool: Understand what makes an effective prompt by comparing your original with the improved version

Auto-Improve is now available in beta for all Helicone users. Try it today with one of your prompts!

We now support Anthropic’s Prompt Caching feature! Monitor and analyze cached prompt tokens and costs alongside regular requests through Helicone’s observability tools. Enable prompt caching through your existing Anthropic integration with Helicone - no additional configuration needed.

• Learn more about integrating with Anthropic in our Anthropic integration guide.
• Learn more about Anthropic Prompt Caching in Prompt caching with Claude.

Perplexity AI + Helicone

We’re excited to announce our integration with Perplexity AI, bringing powerful observability tools to your Perplexity model implementations! Get started with just two simple steps:

1. Generate a write-only API key in your Helicone account.
2. Update your Perplexity AI base URL to:

   https://perplexity.helicone.ai
   

That’s all it takes! Now you can monitor, analyze, and optimize your Perplexity AI models with Helicone’s comprehensive insights.

For more details, check out our Perplexity AI integration guide.

Nebius AI + Helicone

We’re excited to announce our integration with Nebius AI, bringing powerful observability tools to your Nebius model implementations! Get started with just two simple steps:

1. Generate a write-only API key in your Helicone account.
2. Update your Nebius AI base URL to:

   https://nebius.helicone.ai
   

That’s all it takes! Now you can monitor, analyze, and optimize your Nebius AI models with Helicone’s comprehensive insights.

For more details, check out our Nebius AI integration guide.

Elevate your prompt development with our new best-in-the-world editor, designed for peak efficiency and creativity. Now Live for all Prompts & Experiments users!

Smart Editing Capabilities

• Advanced Text Editors

  • Intelligent auto-complete
  • Magic Toolbar with powerful shortcuts
    • ⌘ E: Quickly add variables
    • ⌘ J: Insert XML delimiters
    • ⌘ K: Perform instant edits

• Workflow Optimization

  • Side-by-side prompt-response view
  • Instant Save & Run for real-time feedback

Comprehensive Control & Flexibility

• Version Management

  • Fast version switching
  • One-click version promotion
  • Simple deployment via prompt ID

• Data & Model Support

  • Import variables from live production data
  • Quickly link to an experiment
  • Message pairs and prefill support
  • Full compatibility with leading AI models:
    • Anthropic
    • OpenAI
    • Google
    • Meta
    • DeepSeek
    • And more!

DeepSeek AI + Helicone

We’re excited to announce our integration with DeepSeek AI, bringing powerful observability tools to your DeepSeek model implementations! Get started with just two simple steps:

1. Generate a write-only API key in your Helicone account.
2. Update your DeepSeek AI base URL to:

   https://deepseek.helicone.ai
   

That’s all it takes! Now you can monitor, analyze, and optimize your DeepSeek AI models with Helicone’s comprehensive insights.

For more details, check out our DeepSeek AI integration guide.

User Histograms: Analyze LLM Usage Patterns

We’re excited to introduce User Histograms, a powerful new visualization tool that helps you understand user behavior patterns across your LLM applications.

Key Features

• Distribution Visualization: See how your users are distributed across different metrics like token usage, costs, and request volumes
• Percentile Analysis: Quickly identify power users and understand usage patterns at different percentiles
• Interactive Filtering: Filter and segment your user data to focus on specific time periods or user groups

Use Cases

1. Usage Pattern Analysis

   • Identify usage clusters and understand how different user segments interact with your LLM applications
   • Spot outliers and investigate unusual usage patterns

2. Cost Optimization

   • Understand cost distribution across your user base
   • Make informed decisions about pricing tiers and usage limits

3. Capacity Planning

   • Analyze token usage patterns to better predict and plan for scaling
   • Understand peak usage patterns across your user base

To access User Histograms, navigate to the Users tab in your Helicone dashboard and click on the Histograms view.

🎉 Experiments is here!

We are thrilled to announce that Experiments is out of beta.

Experiments is designed to help you tune your LLM prompt, test it on production data, and verify your iterations with quantifiable data.

Main use cases

1. Continuous Improvement

Analyze production edge cases to refine your application’s performance.

2. Pre-deployment Testing

Benchmark new releases rigorously before rolling out to production environments.

3. Structured Testing

Implement LLM-as-a-judge or custom evaluation metrics, then compare prompt variations side-by-side with quick, actionable feedback loop.

4. Prompt Optimization

Determine the best prompt for production by running evaluators to prevent performance regressions.

For detailed documentation, refer to our updated docs.

We’re excited to announce support for tracking AWS Bedrock models requests through Helicone

How to track your requests?

To track your Bedrock requests through Helicone, you can set the Bedrock client’s endpoint to use the Helicone Proxy.

endpoint="https://bedrock.helicone.ai/v1/<region>"

For detailed API documentation, please refer to our updated docs.

Cerebras Integration

We’re excited to announce the addition of Cerebras as a new model provider on our platform. This integration expands our suite of available AI models and provides more options for our users.

Getting Started

To start using Cerebras models, create an account on Cerebras and then create a new API key. Once you have your API key, you can add it to your Helicone configuration as a base_url.

base_url="https://cerebras.helicone.ai/v1"

For detailed API documentation, please refer to our updated docs.

Webhooks: Real-Time Integration and Automation

We are excited to announce the addition of webhooks to our platform, enhancing real-time integration and automation capabilities. With this update, you can:

• Set up and monitor webhooks for real-time data processing.
• Seamlessly integrate webhook routes with your applications.
• Test webhooks locally using tools like ngrok.
• Randomly sample webhook events and filter them by specific properties to tailor your data processing needs.
• Utilize webhooks for evaluations and score tracking, enabling more precise and automated performance assessments.

For detailed instructions, please refer to our Webhooks Setup Guide.

Prompt UI Refresh

We’ve refreshed the Prompts interface to align with our new UI style — now simpler, more productive, and consistent throughout. Key improvements include:

• Full-width interface
• Ability to view production inputs in a prompt template
• Easily rollback to a previous prompt
• Ability to edit prompts and push to production from the UI

Check it out in the Prompts tab in Helicone!

New Claude 3.5 Sonnet (claude-3-5-sonnet-20241022-v2): Full Cost Support and Tracking

We’re excited to announce immediate support for Anthropic’s latest Claude 3.5 Sonnet model (claude-3-5-sonnet-20241022-v2), released in October 2024.

What’s New in This Version?

• Improved performance across various tasks
• Enhanced capabilities in analysis, coding, and creative writing
• Better contextual understanding and response relevance

Performance Tracking and Cost Management

Our platform now offers:

• Comprehensive performance tracking for claude-3-5-sonnet-20241022-v2
• Updated pricing calculator for accurate cost estimation

How to Use

Refer to our Anthropic Integration Guide for details on how to use the new model with Helicone

Learn More About Claude 3.5 Sonnet

Discover Helicone’s experiments, a new spreadsheet-like interface designed for efficient LLM prompt experimentation. Easily manage multiple prompt variations, run flexible experiments, and gain data-driven insights to optimize your AI prompts.

Get early access now 👉 helicone.ai/experiments

Redesigned Requests Page for Enhanced LLM Observability

We’re excited to announce a major redesign of our Requests page, enhancing the user experience and efficiency for AI LLM observability.

Key Improvements

• Streamlined Navigation: Quick toggle between requests without closing the drawer, allowing for faster review and comparison.
• Compact Information Display: More data visible at a glance with a sleeker, more compact row design.
• Reduced Visual Clutter: A cleaner interface that focuses on essential information.
• Enhanced Time Selector: Improved configuration options and quick select features for more precise data filtering.
• Unobstructed Page Navigation: Chat widget no longer blocks page navigation, ensuring a smoother user experience.

Benefits for LLM Developers and Data Scientists

• Efficient Prompt Analysis: Easily view and compare prompts across multiple requests.
• Improved Performance Monitoring: Quickly identify trends and anomalies in your LLM applications.
• Streamlined Workflow: Navigate through large volumes of request data with ease.

This redesign reflects our commitment to providing the best tools for AI LLM observability. We’ve focused on enhancing the core features that matter most to our users, making it easier than ever to gain insights from your LLM application data.

We encourage you to explore the new Requests page and experience the improvements firsthand. Your feedback is valuable as we continue to refine and enhance Helicone’s observability platform.

We are thrilled to announce the addition of two essential npm packages: @helicone/async and @helicone/helpers. Additionally, we are also deprecating the @helicone/helicone package.

Why These Changes?

• Optimized Package Size: The previous @helicone/helicone was wrapped around OpenAI, resulting in a bulky package size.
• Enhanced Function Utilization: Many functions within the old package were unused and outdated. The new approach ensures that only necessary functions are included and are up to date.

Detailed Changes

• Deprecated @helicone/helicone:

  • This package is officially deprecated and will no longer receive updates.
  • Existing functions within this package will continue to operate as expected to ensure a smooth transition.

• Added @helicone/async:

  • HeliconeAsyncLogger Class: Previously part of @helicone/helicone, this class is now housed within @helicone/async. It retains all existing functionalities, offering robust asynchronous logging capabilities.

• Added @helicone/helpers:

  • HeliconeManualLogger Class: Moved from @helicone/helicone to @helicone/helpers, this class now adopts a more functional approach. Visit the docs to learn more.
  • Enhanced Features:
    • Vector Database Support: Added request logging support for vector databases.
    • External Tools: You can also log your external tool calls.

Summary Reports

Get weekly summary reports of your LLM usage

We’ve launched a new feature that keeps you updated on your LLM usage with detailed weekly reports delivered directly to your inbox every Monday at 10 AM UTC. These reports provide a comprehensive overview of key metrics, including total usage, cost analysis, number of requests, error rate, active users, threats, number of sessions, and average session costs.

With these automated reports, you can easily monitor your AI performance, optimize your usage, and make data-driven decisions for your projects. Ensure you’re staying on top of your LLM utilization and maximizing the value of your resources.

Ready to get started? Configure your weekly summaries now.

O1 Models: Support Added with Token and Cost Tracking

Immediate Support for OpenAI’s o1 Models

We’re excited to announce support for OpenAI’s new o1 models, along with comprehensive tracking of token counts and spending.

What Are o1 Models?

OpenAI’s o1 models represent a significant advancement in language AI. They use reinforcement learning to perform complex reasoning tasks, generating an internal chain of thought before producing a final response. This leads to enhanced performance and new capabilities for your applications.

Accurate Cost Tracking

Our platform now fully supports cost tracking for o1 model usage. Due to the unique way these models process information, it’s important to provide token counts for both input and output to ensure accurate cost calculations.

How to Ensure Accurate Tracking

• Using Integrations: If you’re using integrations like Langchain, LlamaIndex, or LiteLLM, token usage is automatically tracked.
• Streaming Usage: For accurate cost calculation while streaming, refer to our guide on Correct Cost Calculation While Streaming.

Learn More About o1 Models

Streamline your AI data organization and analysis with Helicone’s new Datasets feature. Designed for LLM developers and data scientists, this tool simplifies data handling for improved AI model performance.

Key Features of Helicone Datasets:

1. Dataset Creation: Quickly set up and organize your AI training data within the requests page.
2. Export: Easily export your data as JSONL for training or finetuning.
3. Edit: Edit your dataset and save it as a new version.

Benefits for AI Development:

To begin using the Datasets feature:

1. Navigate to the Requests page in your Helicone dashboard.
2. Enter select mode by clicking the select icon in the top right corner.
3. Select the data points you want to include in your dataset.
4. Click on “Create Dataset” and give it a name.
5. Access your datasets from the new Datasets tab to export or edit as needed.

Collapsible Sidebar

Enhance your workflow with our new collapsible sidebar feature. Users can now easily toggle the sidebar visibility, maximizing screen real estate and improving focus. This update offers:

• One-click sidebar collapse/expand
• Increased workspace flexibility
• Improved screen space utilization
• Seamless transition between full and minimized views

Optimize your productivity by customizing your interface on demand. Experience a cleaner, more adaptable workspace with our latest sidebar enhancement.

Slack Alerts

Real-Time Alerts Now Available in Slack for Faster Issue Resolution

Stay on top of critical issues with Helicone’s latest update: Slack Integration for Alerts. In addition to email notifications, you can now receive real-time alerts directly in your Slack workspace for faster action when something goes wrong.

To get started, visit the Alerts page to create or edit an alert. Enhance your team’s productivity by responding to key notifications without delay.

#1 Product of the Day on Product Hunt

Helicone Reaches #1 on Product Hunt!

This achievement reflects our team’s hard work and the incredible support from our community. We’re thrilled about the boost in visibility for our platform!

Highlights:

• #1 on Product Hunt’s daily leaderboard
• Positive feedback from the open-source community
• Surge in new user sign-ups and engagement

A huge thank you to everyone who upvoted, commented, and shared Helicone. Your support motivates us to keep improving!

For more on our Product Hunt journey, check out our blog posts:

• How to Automate a Product Hunt Launch - Lessons from Helicone’s Success
• Behind 900 pushups, lessons learned from being #1 Product of the Day

Links:

Product Hunt: Helicone on Product Hunt

Docker images on Docker Hub

Docker images now available on Docker Hub We’ve started publishing Docker images on Docker Hub.

This update simplifies Helicone deployment on platforms that don’t natively support the Google Container Registry. For detailed instructions, please refer to our updated self-hosting guide.

Links:

Docker Hub: helicone

New hpstatic Function for Static Prompts in LLM Applications

We’ve added a new hpstatic function to our Helicone Prompt Formatter (HPF) package. This function allows users to create static prompts that don’t change between requests, which is particularly useful for system prompts or other constant text. The hpstatic function wraps the text in <helicone-prompt-static> tags, indicating to Helicone that this part of the prompt should not be treated as variable input.

Here’s a quick example of how to use hpstatic:

import { hpf, hpstatic } from "@helicone/prompts";

const systemPrompt = hpstatic`You are a helpful assistant.`;
const userPrompt = hpf`Write a story about ${{ character }}`;

const chatCompletion = await openai.chat.completions.create(
  {
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: userPrompt },
    ],
    model: "gpt-3.5-turbo",
  },
  {
    headers: {
      "Helicone-Prompt-Id": "prompt_story",
    },
  }
);

This new feature enhances our prompt management capabilities, allowing for more flexible and efficient prompt structuring in your applications.

Ragas Integration for RAG System Evaluation

We’re excited to announce our integration with Ragas, an open-source framework for evaluating Retrieval-Augmented Generation (RAG) systems. This integration allows you to:

• Monitor and analyze the performance of your RAG pipelines
• Gain insights into RAG effectiveness using metrics like faithfulness, answer relevancy, and context precision
• Easily identify areas for improvement in your RAG systems

Check out this quick video overview of the Ragas integration:

To get started with the Ragas integration, visit our documentation for step-by-step instructions and code examples.

Optimistic Updates & Asynchronous Loading in Requests Page

We’ve improved data loading in the Requests page of the Helicone platform. By fetching metadata and request bodies separately and loading data asynchronously we’ve reduced the time it takes to render large tables by almost 6x, improving speed and UX.

New Assistants UI Playground

We’re thrilled to announce a major update to our Assistants UI Playground! Head to the Playground and click the “Try New Playground” button to explore the latest improvements:

• Streamed responses for real-time interaction
• Enhanced tool rendering for better visualization
• Improved reliability for a smoother experience

Coming soon:

• Expanded model support
• Advanced prompt management
• Integrated Markdown editor

Try out the new Playground today and elevate your LLM testing experience!

Fireworks AI + Helicone

We’re excited to announce our integration with Fireworks AI, the high-performance LLM platform! Enhance your AI applications with Helicone’s powerful observability tools in just two easy steps:

1. Generate a write-only API key in your Helicone account.
2. Update your Fireworks AI base URL to:

   https://fireworks.helicone.ai
   

That’s all it takes! Now you can monitor, analyze, and optimize your Fireworks AI models with Helicone’s comprehensive insights.

For more details, check out our Fireworks AI integration guide.

Dify + Helicone

We’re thrilled to announce our integration with Dify, the open-source LLM app development platform! Now you can easily add Helicone’s powerful observability features to your Dify projects in just two simple steps:

1. Generate a write-only API key in your Helicone account.
2. Set your API base URL in Dify to:

   https://oai.helicone.ai/<API_KEY>
   

That’s it! Enjoy comprehensive logs and insights for your Dify LLM applications.

Check out our integration guide for more details.

Prompts package

We’re excited to announce the release of our new @helicone/prompts package! This lightweight library simplifies prompt formatting for Large Language Models, offering features like:

• Automated versioning with change detection
• Support for chat-like prompt templates
• Efficient variable handling and extraction

Check it out on GitHub and enhance your LLM workflow today!