Intro

In 2023, the NEAR ecosystem navigated a challenging market landscape while still shipping key technical improvements and furthering its mission of widespread Web3 adoption with several notable applications boasting over a million daily active users during different periods of the year. Helping to drive these user growth numbers has been the rollout of the Blockchain Operating System (BOS) and the introduction of FastAuth. The BOS, launched in Q1 2023, streamlines the creation and deployment of decentralized applications (DApps) by abstracting complex blockchain layers while FastAuth enables rapid user account creation with minimal barriers.

However, the ecosystem has not just been focused on user interface improvements. Q4 2023 saw several announcements and partnerships that look to strategically align NEAR with the Ethereum blockchain but also with the broader modular ecosystem, writ large. NEAR announced collaborations to integrate its technology with both Polygon Labs and Eigen Labs, two projects looking to scale Ethereum in different ways, by contributing to its modular theses. These initiatives aim to leverage NEAR's capabilities in zero-knowledge proofs and data availability, presenting a cost-effective alternative for Layer 2 solutions. In this report, we will take the time to discuss the scalability issues hampering Ethereum and its rollups, uncover how NEAR has emerged as a solution, and dissect what makes Near DA distinct from other solutions. 

The Modular Movement 

Increasing block size and reducing block time seems a straightforward way to enhance scalability, but it poses a significant risk of centralizing the network. Rapidly expanding the blockchain's ledger size escalates hardware requirements for running a full node, potentially limiting the number of participants able to maintain these nodes. This situation undermines the fundamental principle of decentralization in blockchain technology. To avoid this scenario, many blockchains have decided to scale in a more “modular” approach to sustainably scaling blockchains while maintaining low validator requirements and attempting to preserve the decentralization of the network. So, what is this modular design? How does data availability factor into it?

“Modular” blockchains differentiate themselves from “integrated” or “monolithic” chains by separating the essential functions of a network, such as execution, consensus, settlement, and data availability, into different layers. Integrated chains handle all of these elements while a modular approach segments out the following layers: 

• Execution Layer: This is the initial stage where transactions and changes in the blockchain's state are processed. It is the layer with which users commonly interact, and transaction computation occurs.
• Settlement Layer: The settlement layer verifies the execution of transactions and adjudicates disputes. Notably, this layer is absent in monolithic chains and represents an optional component within the modular structure of the blockchain.
• Consensus Layer: The consensus layer is pivotal in maintaining the order and finality of transactions within the blockchain. It involves a network of full nodes that download and execute block contents, subsequently reaching a consensus on the legitimacy of state transitions.
• Data Availability Layer: This layer is responsible for publishing and storing the data needed to validate state transitions. In cases of attacks where malicious block producers might withhold transaction data, the availability of this data for verification is crucial.

These modular systems offer a distinct advantage in terms of flexibility and scalability when compared to traditional monolithic blockchain structures, without compromising on the essential attributes of security and resistance to censorship that are hallmarks of blockchain technology.

By distributing the total workload across various specialized layers, a modular chain effectively utilizes a divide-and-conquer strategy. This approach allows for a higher aggregate throughput than what any individual node could handle on its own. Essentially, it's about optimizing efficiency by allocating specific tasks to specialized components within the blockchain.

Modular blockchains are not some theoretical idea. Rather, many blockchains have begun to adopt this design/strategy. Prominent blockchain networks like Ethereum, Polkadot, Cosmos, Avalanche, Celestia, Polygon, and NEAR are at the forefront of this trend. Each of these networks is pursuing unique strategies to scale their own networks but with the same general idea of dividing the blockchain’s workload among different layers. 

What is the Data Availability (DA) Problem?

The Data Availability (DA) “Problem” only really became a “problem” once blockchains began to experiment with scaling solutions like L2s and rollups. Pre-2021 (roughly speaking), data availability (DA) for most blockchains was not a concern for two reasons: one, most blockchains did not have enough usage and thus had no reason to scale, and, two, the monolithic approach of each (full) node downloading the entire block to check availability was adequate. However, as Ethereum began to struggle to keep up with block space demand, L2s became the accepted solution for scaling computation. But while this solved one issue, another one, the DA Problem, arose.

The primary role of the Data Availability (DA) layer is to meticulously capture and record transaction data, guaranteeing its full accessibility, verifiability, and utility, thereby circumventing any issues related to data availability. The DA layer is constituted by a variety of individual blockchains, each aimed at reducing the expenses associated with data for Layer 2 solutions and independent application-specific blockchains. Presently, Ethereum's Layer 1 stands as the preeminent network for both data availability and transaction settlement. Nevertheless, Layer 2 networks incur significant costs when integrating their data with the Ethereum Layer 1 platform, a process that ensures data transparency and auditability. This scenario has paved the way for innovative data availability platforms like Celestia, Eigen DA, and Near DA to make their mark in the technology landscape. These emerging solutions offer Layer 2 networks and application-specific chains the opportunity to slash their data availability expenses dramatically, in some instances by as much as 95%, while still maintaining the capability to finalize transactions through the Ethereum Layer 1 network.

Ethereum Rollups and DA

Layer 2 rollups work by combining multiple transactions into a single rollup block, which is then added to the Layer 1 chain, reducing the burden on the main network's resources. This approach enables a significant increase in the number of transactions handled and opens doors for better scalability.

Data availability plays a vital role in ensuring the effectiveness and security of these off-chain scaling solutions. It involves making transaction data easily accessible and verifiable, allowing other nodes in the network to uphold trust and integrity in how the system operates. Without proper measures in place, rollups could turn into isolated data storage areas that are at risk of being manipulated or experiencing financial losses.

The responsibility of confirming transaction data falls upon full nodes operating on Layer 1 chains. In a blockchain network, there are generally two types of nodes: full nodes and light clients. Full nodes, also known as fully validating nodes, download and validate every transaction in the blockchain. This process is resource-intensive but offers the highest security level as these nodes are not susceptible to accepting blocks with invalid transactions. On the other hand, light clients, suitable for less resourceful computers, only download the block header and rely on the assumption that the block contains valid transactions, making them less secure than full nodes.

A block header is essentially the metadata for the block, including the Merkle root of transactions. The transaction data forms the bulk of the block and includes the actual transactions. 

‍

A key solution to bolstering the security of light clients involves the use of fraud proofs. These are mechanisms by which full nodes can inform light clients of any invalid transactions within a block. These proofs enable light nodes to detect malicious behavior, such as the inclusion of an invalid transaction by a block producer, without downloading the entire block. However, this solution has its limitations.

A significant issue arises if block producers publish only the block header without the transaction data. In such cases, full nodes cannot validate transactions or generate fraud proofs for invalid ones. The data availability problem arises when essential transaction data is omitted, causing light nodes to diverge from full nodes and follow an invalid blockchain. This is the “DA Problem” and can lead to users’ potential loss or freezing of funds. Rollup networks address this by mandating on-chain storage of transaction data but face limitations in speed and fees due to reliance on low-scalability platforms like Ethereum.

Therefore, it's essential for block producers to publish all the data of their blocks, and mechanisms must be in place to ensure this compliance. However, as scaling occurs and more transactions are conducted off-chain, these nodes face the challenge of checking the availability and accuracy of such data without becoming bottlenecks themselves. This task becomes even more difficult when considering the importance of maintaining decentralization.

No Data, No Proof Verification

In the current design, rollups only work so long as complete execution data is accessible on the mainnet. At its core, the DA Problem originates because full nodes need all the data in order to verify accuracy (checking proofs) whereas light clients simply check block headers. This dependence by full nodes reveals a vulnerability: a malicious block producer might opt to publish only block headers, withholding partial or complete data. This obstructs full nodes from verifying transactions and identifying invalid ones, consequently impeding the generation of fault proofs. This scenario, while easily detectable by full nodes that can download and scrutinize the entire block, poses a significant challenge for light clients. Light clients, which do not download the full block and inherently trust state transitions, may inadvertently track headers of an invalid chain, diverging from full nodes.

The problem of data availability is pertinent to both fraud and validity-proof-based systems. Without full access to data, honest full nodes are incapable of generating the necessary proofs to validate or invalidate a block. This predicament manifests differently in fraud-proof and validity-proof-based systems:

• Fraud-proof System: Light nodes track headers of a potentially invalid chain, making it impossible for honest nodes to verify the blocks.
• Validity Proof-based System: Light nodes follow a chain with a valid yet undisclosed state, preventing honest nodes from producing new blocks. This stagnation effectively grants attackers control over the chain, similar to having custody of everyone's funds.

In both cases, light nodes, due to their limited scope, fail to detect the issue and unintentionally diverge from the path of full nodes.

This situation leads to a pivotal realization that irrespective of the enhancements made to the execution layer or the implementation of various proof mechanisms, the throughput of rollups is ultimately constrained by data availability. Once the L1's data capacity is fully utilized by the rollup, a bottleneck is reached, preventing the processing of additional transactions.  The greater the data capacity of the L1, the higher the potential throughput for the rollups. As a result, the primary constraint for a blockchain's scalability has shifted to the realm of data availability. Addressing this bottleneck is crucial for advancing the scalability and reliability of blockchain networks.

DA as a Bottleneck

The primary operations of rollups, including transaction processing and storage updates, occur off-chain, thus incurring minimal costs. Currently, rollups transmit batches of compressed transactions to Ethereum L1 as calldata, which is the information contained in the transaction’s Data field. The cost associated with writing this calldata to Ethereum is the primary expense for rollups, with ~70-90% of their revenue used to post calldata. 

Every byte of data on the Ethereum network requires 16 units of gas for processing. Transactions involving ETH transfers consume 12 bytes, with more complex smart contract transactions requiring much more. Given the current target gas limit of 15 million for each block, this translates to a theoretical maximum throughput of approximately 6,000 transactions per second (TPS). This upper limit is attainable only if the entire capacity of Layer 1 block space is dedicated exclusively for handling Layer 2 data.

Cost efficiency remains a crucial factor for users and developers. As of January 2024, Ethereum's rollup solutions, Optimism and Arbitrum, offer transaction costs of $0.09 and $0.08 respectively for sending ETH, and $0.17 and $0.24 for swaps. While these costs are significantly lower than those on Ethereum's mainnet, they are still notably higher compared to next-generation L1s like Solana and Aptos. Because posting data to Ethereum mainnet remains rollup operators’ biggest cost, multiple solutions are in the works.

Proto-danksharding (PDS), expected to launch on Ethereum mainnet in Q1 2024, should help to alleviate the costs and constraints associated with calldata. Under PDS, rollups could post bundles under a new “blob” transaction type, which is much cheaper than calldata. Blob transactions can enable up to ~1MB average per block for data storage as opposed to the 10KB currently with calldata. 

Rollup transactions would have their own “channel,” operating through a novel data blob market that uses its own fee structure and floating gas limits. This means that even with heightened demand and activity from DeFi or NFTs, data costs won’t go up for rollups. This creates two different gas markets - one for general computation and one specifically for data availability (DA), making the overall economic model more efficient than it was previously.

While PDS can help ETH rollup fees with a native solution, modular DA layers have surfaced as an alternative solution that can potentially slash DA costs even more (by up to 99%).

Announcing NEAR DA

In Q4 2023, the Near Foundation announced its plans to position itself as a key player in the blossoming data availability ecosystem that aims to alleviate the increasing demands (and costs) associated with storing data on the Ethereum blockchain. The announcement of NEAR DA, spearheaded by the Near Foundation, comes at a critical time when numerous projects are racing to launch their own DA solutions, including Celestia, EigenDA, Avail, and others. Early indications are that NEAR DA can drastically reduce costs associated with data posting on the blockchain, with estimates suggesting orders of magnitude reduction in cost.

How Does NEAR DA Work?

Key to NEAR DA’s architecture is the Blob Store Contract, which serves as a storage mechanism for arbitrary Data Availability (DA) blobs. Due to the relatively affordable state storage costs on the NEAR blockchain—100KiB for one NEAR —the system is designed to optimize storage efficiency by not housing the blob data directly within the blockchain state. Storing the same volume of calldata on the NEAR Protocol is ~85,000 times more cost-effective compared to Ethereum. Additionally, a trustless off-chain light client designed for NEAR facilitates straightforward validation of on-chain storage of rollup data.

‍

As of December 2023, 231 kB of calldata on NEAR costs $0.0016, while the same calldata on Ethereum L1 costs users $140.54, and on Celestia costs users $0.046.

‍

The Blob Store Contract operates under several principles:

• Consensus Assurance: NEAR validators provide consensus around the submission of a blob.
• Data Storage: The function input data is temporarily retained by full nodes.
• Long-Term Archiving: Archival nodes and indexers facilitate longer-term data retention.
• Optimized Consensus Load: The system avoids overburdening the consensus mechanism with excessive data.
• Accessible Commitment: The blob commitment process is designed to be simple and accessible for a wide range of users.

This efficiency is achieved by leveraging the consensus mechanism around receipts within the NEAR protocol. When a chunk producer processes a receipt, consensus is achieved. Following the processing and inclusion of the chunk in a block, the receipt is no longer necessary for consensus and can thus be pruned after a minimum duration of three NEAR epochs, approximately five epochs in practice. The responsibility for maintaining the transaction data post-pruning falls to archival nodes, although the data can also be accessed from indexers.

To ensure the integrity and origin of the blob data, the system utilizes a blob commitment mechanism, which is currently under optimization. This commitment is generated by dividing the blob into 256-byte chunks, hashing each with SHA-256, and then constructing a Merkle tree from these hashes. The root of this Merkle tree becomes the blob commitment, a concise 64 bytes when combined with the transaction ID, ensuring a verifiable, compact representation of the data.

Complementing the Blob Store Contract is a light client designed for trustless off-chain operations with DA-enabled features like KZG commitments (not required but optional), Reed-Solomon erasure coding, and storage connectors. This light client enables users to access transaction and receipt inclusion proofs efficiently, which is crucial for verifying the legitimacy of blobs or confirming their submission to NEAR.

Near DA Adoption

In a significant development for the modular blockchain ecosystem, the NEAR Foundation and Polygon Labs have jointly announced the integration of NEAR DA with the Polygon CDK, a toolkit designed for the creation of customized, ZK-powered Layer 2 solutions. NEAR DA will reduce operational costs and enhance scalability, while the Polygon CDK enables developers to build custom Layer 2 chains that are fine-tuned to their specific needs. These customizable features include transaction costs, native token selection, throughput capabilities, and more. By aligning with the Polygon ecosystem, developers gain access to a network of blockchain solutions, including Polygon PoS, Polygon zkEVM, and other Polygon CDK-enabled chains. The integration with NEAR DA expands the capabilities of Polygon CDK, enabling developers to leverage a comprehensive, modular data availability solution right out of the box. This partnership is particularly significant as it marks the first instance of NEAR DA's integration with a ZK-based Layer 2 stack, broadening the horizons for developers in search of scalable data availability solutions. 

Additionally, NEAR DA integration was announced for Arbitrum Orbit, an advanced platform enabling developers to create bespoke rollups using Arbitrum’s underlying technology. Arbitrum Orbit serves as a scalable L2/L3 solution for Ethereum, providing developers with the tools to construct unique chains tailored to their specific needs. 

2024

Looking ahead to 2024, the NEAR Protocol team aims to deliver a suite of significant enhancements beyond simply serving as a DA solution. Some top goals for 2024 include improving the protocol's usability, scalability, and decentralization. One of the key initiatives is Stateless Validation, a project that reimagines the Nightshade sharding design by integrating state in memory, thereby eliminating the need for fraud proofs and substantially enhancing shard performance. This critical development is scheduled for completion in Q2 2024 and is expected to not only fulfill the objectives of Phase 2 of sharding but also significantly bolster the throughput of each shard to support NEAR's growing usage. 

Another innovative feature in the pipeline is Account Aggregation, which will allow users to control accounts across multiple chains using a singular NEAR account. This feature necessitates the incorporation of chain signatures and the introduction of a new runtime API to handle the asynchronous nature of these signatures. 

In another collaboration with the Polygon team, the NEAR Protocol is also advancing its work on zkWASM. The goal here is to develop a prover for WebAssembly smart contracts, which could play a crucial role in validating NEAR's state transitions using zero-knowledge proofs and offer an alternative execution environment for Layer 2 solutions. As part of this alliance, the NEAR Foundation has positioned itself as a core contributor to Polygon's Chain Development Kit (CDK).

Finally, the team is conducting research on the future of sharding, focusing on two main approaches: synchronous sharding and ZK-centric sharding. Synchronous sharding is geared towards enabling the synchronous execution of smart contracts, thereby improving the developer experience, while ZK-centric sharding leverages zero-knowledge proofs for validating state transitions across different shards, enhancing scalability and decentralization. Moreover, there is an ongoing exploration into the final phase of sharding, which involves dynamically adjusting the number of shards in response to network usage. Collectively, these initiatives represent NEAR Protocol's commitment to fostering a more robust, user-friendly, and scalable blockchain ecosystem.

Conclusion

2023 marked a significant milestone in the evolution of blockchain technology, with the NEAR ecosystem emerging as a key contributor to the new modular era as well as the general Web3 space.  NEAR's introduction of the Blockchain Operating System (BOS) and FastAuth has been instrumental in simplifying the user experience, leading to an impressive surge in daily active users. The BOS, with its ability to abstract complex blockchain layers, and FastAuth's facilitation of rapid account creation have collectively lowered the entry barriers for users and developers alike.

However, the true testament to NEAR's commitment to blockchain innovation is its strategic alignment with the Ethereum ecosystem and its involvement in the broader modular blockchain movement. The collaboration with Polygon Labs highlights NEAR's dedication to interoperability and its role in scaling Ethereum in unique ways. By contributing its expertise in zero-knowledge proofs and data availability, NEAR is presenting itself as a formidable player in offering cost-effective alternatives for Layer 2 solutions.

The modular approach, which involves segmenting key blockchain functions into different layers, has proven to be a game-changer in maintaining network decentralization while enhancing scalability. This approach allows for more flexible and efficient network operation without compromising the core principles of security and resistance to censorship inherent in blockchain technology. NEAR's role in promoting this modular design philosophy is not just innovative but also necessary for the sustainable growth of blockchain networks.

The NEAR DA initiative, announced in Q4 2023, is perhaps one of the most significant strides in this direction. This groundbreaking solution, aimed at reducing the costs associated with data posting on the Ethereum blockchain, showcases NEAR's capability to offer practical and cost-effective solutions to the data availability challenges faced by Layer 2 networks. The Blob Store Contract, a central component of NEAR DA, optimizes storage efficiency and demonstrates NEAR's commitment to providing scalable and affordable data solutions.

Moreover, the integration of NEAR DA with Polygon CDK marks a remarkable collaboration in the blockchain space. This partnership not only enhances the scalability and cost-efficiency of blockchain operations but also opens up new possibilities for developers to create customized, ZK-powered Layer 2 solutions. The combination of NEAR's data availability solution and Polygon's toolkit represents a significant step forward in the development of modular blockchain ecosystems.

‍

‍

‍

‍

Disclaimer: This report was commissioned by the NEAR Foundation. This research report is exactly that — a research report. It is not intended to serve as financial advice, nor should you blindly assume that any of the information is accurate without confirming through your own research. Bitcoin, cryptocurrencies, and other digital assets are incredibly risky and nothing in this report should be considered an endorsement to buy or sell any asset. Never invest more than you are willing to lose and understand the risk that you are taking. Do your own research. All information in this report is for educational purposes only and should not be the basis for any investment decisions that you make.

‍