Dmitry

October 31, 2024 — 7 min read

Solana Mainnet has recently been settling over a billion non-vote transactions every month. With Solana's high throughput and short block times, accessing real-time block and transaction data efficiently can give you a competitive edge. Managing this volume of data, however, can be challenging, especially when your application needs real-time information for time-sensitive operations. QuickNode's suite of Solana tooling offers multiple approaches to access the data you need, each optimized for different use cases. Choosing the right approach to handle this data flow is critical to ensuring your application performs reliably at scale.

Solana Data Solutions: Understanding Your Options

QuickNode offers three approaches for accessing real-time Solana data:

1. Traditional WebSocket Subscriptions: Direct connections to Solana nodes for real-time updates, ideal for simple applications and rapid development.

2. Yellowstone gRPC: High-performance, extremely fast, typed interface built on Solana's Geyser plugin system, offering enterprise-grade reliability with granular control.

3. QuickNode Streams: Managed solution for processing and routing Solana data to multiple destinations, with built-in filtering and historical data support.

Each option serves different needs and offers unique advantages. Let’s dive into when and why you might choose one over the other.

Traditional WebSocket Subscriptions

WebSockets provide the most straightforward way to receive real-time updates from the Solana blockchain. They maintain a persistent connection to a Solana node, allowing you to subscribe to specific events like account changes, program logs, or new blocks with the JSON RPC API. Under the hood, WebSockets use a bi-directional communication protocol that keeps a single TCP connection open between your client and the node (more information on WebSockets: Mozilla). This approach offers a lightweight solution for tracking specific accounts or programs with minimal configuration complexity.

Best suited for:

• Simple applications monitoring specific accounts or programs

• Development and testing environments

• Web applications

While WebSockets offer simplicity and direct access, they require careful management of connection stability and reconnection logic in production environments. They're an excellent choice for getting started or for applications with straightforward real-time data requirements.

Yellowstone gRPC Geyser Plugin

Yellowstone leverages Solana's Geyser plugin system to provide enterprise-grade data access without burdening validators. By using gRPC - Google's high-performance framework - Yellowstone offers extremely low-latency streaming of account updates, transactions, and blocks directly from validator nodes (more information on gRPC: grpc.io). This architecture allows for efficient, type-safe communication that's particularly valuable for applications where performance and reliability are critical.

Best suited for:

• High-frequency trading systems requiring minimal latency

• Custom indexing solutions needing direct validator access

• Enterprise applications requiring fine-grained control

• Teams with existing gRPC infrastructure

While Yellowstone requires more setup than WebSockets or Streams, it provides high-performance data access for latency-sensitive applications. Its direct integration with validator infrastructure through the Geyser plugin system makes it particularly well-suited for teams wanting to maintain their own data pipeline.

Streams

Streams offers a managed solution that simplifies how you handle Solana's high-volume data flow. Rather than maintaining persistent connections or managing infrastructure, Streams lets you configure your data pipeline through an intuitive interface. It processes and routes real-time blockchain data to your chosen destination, including webhooks, S3-compatible storage, and databases like PostgreSQL and Snowflake.

What sets Streams apart is its ability to process both real-time and historical data through the same pipeline. While other solutions require you to handle data processing on your own servers, Streams handles all computation on QuickNode's infrastructure. Streams is highly cost-efficient: because the processing happens on QuickNode’s server-side, you are only billed for the response data you receive. Streams also has a seamless integration into QuickNode Functions, allowing you to transform and filter your data before it reaches its destination, significantly reducing the load on your servers and simplifying your architecture.

Best suited for:

• Teams wanting a managed data pipeline solution

• Applications requiring historical data access

• Projects sending data to multiple destinations (webhooks, databases, cloud storage)

• Services requiring reliable delivery and automatic retries

Streams handles the complexities of data delivery—including automatic retries, reorg handling, and delivery guarantees—allowing you to focus on building application logic rather than managing infrastructure.

Making the Right Choice

When choosing between these solutions, consider the following key questions about your application's needs.

• Do you need historical data or just real-time updates? If historical data access is crucial, Streams offers built-in backfilling capabilities.

• How mission-critical is your data pipeline? If you need enterprise-grade reliability and control, both Yellowstone and Streams offer robust solutions, with Yellowstone providing more direct control and Streams offering managed reliability.

• Where does your data need to go? If you need to route data to multiple destinations or databases, Streams provides out-of-the-box integration with various endpoints. If you're building a custom pipeline, Yellowstone gives you the flexibility to build exactly what you need.

• How much infrastructure do you want to manage? WebSockets are simple but require your own connection management. Yellowstone offers powerful capabilities but needs more setup. Streams handles the infrastructure for you.

The table below breaks down these considerations in detail.

Key Considerations Table

Getting Started

WebSocket

Set up a new WebSocket connection using the Solana JSON RPC. Grab your WSS Endpoint from your QuickNode Dashboard:

Sample Implementation

import { Connection, PublicKey, AccountInfo } from '@solana/web3.js';
async function main() {
    const connection = new Connection(
        'https://example.solana-mainnet.quiknode.com/123456',
        { wsEndpoint: 'wss://example.solana-mainnet.quiknode.com/123456' }
    );
    const ACCOUNT_TO_WATCH = new PublicKey('SOME_ACCOUNT_TO_WATCH');
    const subscriptionId = await connection.onAccountChange(
        ACCOUNT_TO_WATCH,
        (updatedAccountInfo) => handleIncomingData(updatedAccountInfo),
    );
}
async function handleIncomingData(updatedAccountInfo: AccountInfo<Buffer>) {
    // Apply filters or data parsing logic here
}

Resources

• Guide: How to Create WebSocket Subscriptions to Solana using TypeScript

• Documentation: Solana WebSockets

Yellowstone

Navigate to the Yellowstone gRPC Add-on Page and click “Get Started”, or from your Solana endpoint page in your QuickNode Dashboard, click “Add-ons” and find Yellowstone gRPC and click “Explore”.

Sample Implementation

async function main(): Promise<void> {
    const client = new Client(ENDPOINT, TOKEN, {});
    const stream = await client.subscribe();
    const request = createSubscribeRequest();
    try {
        await sendSubscribeRequest(stream, request);
        await handleStreamEvents(stream);
    } catch (error) {
        stream.end();
    }
}

Resources

• Guide: Monitor Solana Programs with Yellowstone gRPC Geyser Plugin

• Documentation: Yellowstone gRPC Plugin

Streams

From your QuickNode Dashboard, click “Streams” > “+ Create Stream”, or click here.

Sample Implementation

function main(data) {
    try {
        const { streamData } = data || {};
        const matchedTransactions = streamData.transactions
            .filter(matchesFilter)
            .map(tx => formatTransaction(tx, streamData));
        return { matchedTransactions };
    } catch (error) {
        return { error: error.message, stack: error.stack };
    }
}

Resources

• Guide: How to Backfill Solana Transaction Data with Streams

• Guide: How to Stream Solana Program Data with Streams

• Documentation: QuickNode Streams

Growing with QuickNode

As we look to the future, it's clear that high-performance blockchain networks like Solana are gaining significant traction. While the technology continues to evolve and face new challenges, the surge in activity we're witnessing signals a growing demand for scalable blockchain solutions.

At QuickNode, we're committed to staying ahead of these trends, ensuring our infrastructure and tools are ready to support developers and projects, regardless of how the blockchain landscape shifts. Beyond our data access solutions, we offer a comprehensive suite of tools designed to help you build and scale efficiently:

• Metis: Enhanced trading capabilities through our Jupiter Trading API.

• Solana Priority Fee API: Real-time fee estimation with priority fee averages based on recent data.

• Lil’ JIT: Atomic transaction bundling ensuring sequential execution and reliability.

• Solana Digital Asset Standard API: Comprehensive suite of RPC methods for seamless digital asset management and querying.

• IPFS: Robust data storage and retrieval via QuickNode's pinning services and dedicated network gateways on the IPFS network.

• Functions: Serverless compute for custom API endpoints.

• Insights:

Whether you're building a DeFi protocol, launching an NFT marketplace, or developing the next generation of Solana applications, QuickNode provides the complete toolkit you need to succeed in web3.

Not a QuickNode customer yet? Sign up for a free account today and explore our full suite of developer tools!