Development across the Avalanche ecosystem is growing rapidly, with new teams, applications, and assets being created every day. To keep the community up-to-speed on the work Ava Labs is doing to support these amazing efforts, we publish this weekly blog to recap our contributions to key technical areas.

This blog also provides information on how developers can get more involved in the Avalanche ecosystem, through programs like bug bounties, developer office hours, and career opportunities. Without further ado, here is this week’s engineering update:

Apricot Phase Two (AP2) Upgrade Reminder

As alluded to in last week’s engineering update, the code for “Apricot Phase Two: Berlin EIPs & Enhanced Avalanche Native Token (ANT) Support” was released earlier this week. You can read about all the improvements in AP2 here.

AP2 activated at 10 a.m. EDT (2 p.m. UTC) on Wednesday, May 5th on the Fuji Testnet and will activate at 7 a.m. EDT (11 a.m. UTC) on Monday, May 10th on the Avalanche Mainnet. This upgrade includes breaking changes to consensus rules and validators that do not upgrade their node by this time will become unhealthy (as they won’t be able to process blocks produced with AP2 consensus rules) and marked offline by other nodes. Make sure to upgrade!

Networking Optimizations (AvalancheGo v1.4.1 and v1.4.2)

In AvalancheGo@v1.4.0, we added some new message types to support authenticated peer gossiping. Simulations we ran on our internal test networks indicated these new message types wouldn’t meaningfully impact the bandwidth usage of the node. However, nodes that updated to v1.4.0 found that this wasn’t the case. Some node operators reported as high as a 10x increase in upload bandwidth usage.

We ran some additional analysis on our internal networks with data we collected from Avalanche Mainnet and were able to reduce the bandwidth usage of the node to within 10% of v1.3.2 in the v1.4.2 release. We’ve identified a few more optimizations that will roll out in subsequent releases that will bring the bandwidth usage down below that of v1.3.2.

As part of this exercise, we also identified some opportunities to improve stability on low bandwidth devices by reducing bursty network layer usage. In short, nodes with limited bandwidth can, in some cases, saturate their bandwidth by trying to do too much at once, such that all activities time out and nothing gets done (looping endlessly in this state). These changes will roll out with the additional bandwidth optimizations mentioned above.

Preparing for AvalancheGo@v1.4.3: Database Migration

Next week, we plan to release some significant database optimizations for AvalancheGo that will require a database migration. To prevent a drop in connected stake in the network while nodes perform this migration, we’ve developed an automatic upgrade daemon that performs this migration while keeping your existing node running. This daemon will run your validator on the previous version, bootstrap the new database using the v1.4.3 code, and then automatically cut over when complete. The binary orchestration logic developed to implement this daemon will be reused in future database migrations and will be a useful tool for orchestrating custom VMs in the additional subnets a node chooses to participate in.

To minimize network usage, the new database will bootstrap from the locally running node (on the previous version) instead of connecting to the rest of the Avalanche network over the internet. During this process, you will notice an increase in CPU and RAM usage as multiple instances of the node software will be running on your machine. We recommend preparing a bootstrapped volume and moving it onto your node if your machine will not be able to support this increase in CPU and RAM usage. Instructions will be provided on how to do this when the v1.4.3 code is released.

To ensure that validators can roll back in the case of any issue, this migration does not modify any legacy data. Rather, an entirely new database is created that uses an optimized format (this also prevents any race issues during the initial bootstrap process because the v1.4.2 node will still be running). A side effect of this approach is that your disk usage will temporarily double during this process (your machine will have both the old database and the newly formatted database). Once the new database is bootstrapped, you are then free to delete the old database.

In our testing on mainnet, we observed a ~90% reduction in read IO for a validating node. To give you an idea of what this looks like, you can check out the diagram below:

Degraded API Performance + Wallet Downtime

Early this week, the Avalanche non-custodial wallet (wallet.avax.network) encountered some downtime due to API degradation. We posted frequent updates to the new status page (status.avax.network) during the downtime. To stay up-to-date on status updates, make sure to subscribe!

In short, we encountered a ~5x increase in peak API requests in 7 days. Let’s just say this rapid growth pushed the limits of our legacy API architecture (which was still fully deployed at that time on Avalanche Mainnet).

At no time, however, did the Avalanche network stop processing blocks and/or transactions. In fact, there was a pretty large cohort of users interacting with the C-Chain thorough MetaMask throughout the downtime that did not encounter meaningful issues. Some people expressed frustration that the wallet was not available during this downtime. Just a reminder: the wallet source code is open-source and can be run by anyone with their own AvalancheGo if the hosted wallet encounters downtime.

On a different note, I’m proud to share that we began a phased rollout of the new API architecture today and will continue this rollout throughout the weekend. Once fully rolled out, this new architecture should eliminate any rate limiting issues users have encountered. By the end of next week, we also plan to have the performance optimizations we are working on deployed to provide better performance than direct node access for the majority of users. Imagine getting an eth_call, eth_getBlockByNumber, or eth_getTransactionReceipt response in 20ms no matter where in the world you are! If you notice any issues as this rolls out over the next few days, please let us know!

C-Chain Exchange Integration Guide

To better support exchanges integrating with the Avalanche network, we added a new C-Chain exchange integration tutorial to the docs website. We’ve heard stories of people integrating with the C-Chain in 1 day by reusing their previous ETH infrastructure. This will be integrated more deeply into the docs website early next week. If you notice any issues or have any suggestions of how to improve this tutorial, please let us know!

New Websites

This week we launched an updated website for Ava Labs. This new site includes an updated list of the members of our awesome engineering team:

We also created a website specifically for the Avalanche platform:

You can read a blog about this website redesign effort here.

“Behind-the-Scenes” Improvements

Each week many small but meaningful improvements are made to the repos we maintain (usually based on feedback from the Avalanche community. Oftentimes they don’t warrant their own section but are still worth calling attention to. Here are this week’s “behind-the-scenes” improvements:

• Mac binaries for AvalancheGo are now notarized
• Bug fixed when signing large smart contract interactions (like DEX swaps) on Ledger
• New version of Avash released to support new AvalancheGo flags (added in AvalancheGo@v1.4.2)
• Added support for ANT “atomic transfers” in AvalancheJS

Office Hours

Over the last few weeks, the Ava Labs engineering team has been hosting focused office hours on Discord from 2–4 PM ET each Wednesday. Last week the Platform team (AvalancheGo / Coreth) held we hosted office hours for Platform (AvalancheGo / Coreth). This Wednesday, office hours will focus on C-Chain dApps.

If you can’t make the office hours, you can always reach out with questions on Discord or on Twitter.

Join Ava Labs

Ava Labs was founded by Cornell computer scientists who brought on talent from Wall Street to execute their vision. The company has received funding from Andreessen Horowitz, Initialized Capital, and Polychain Capital, with angel investments from Balaji Srinivasan and Naval Ravikant.

We are actively hiring for a number of key technical roles. To ensure we attract the best and brightest to join our team, we support hiring remote candidates from anywhere in the world. If the work we are doing interests you, we’d love to chat! You can apply here.

If you don’t know what role to apply for or want to bump your application to the top of the stack, you can take a short 25 minute technical assessment to demonstrate your skills. We’ll reach out promptly if you demonstrate the talent that would make you a great fit at Ava Labs.

---

Weekly Update (5/7/2021): Ava Labs Engineering was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.