Jul 1, 2024
Summary
On June 10, illicit funds acquired in an exploit on the UwU lending platform were used to open borrow positions on Curve Lend, producing inorganic market dynamics that may have contributed to a liquidation cascade on June 12, causing unexpected consequences for crvUSD. The crux of the adverse effect on crvUSD was a temporary upward depeg due to the unexpected demand for liquidations processing, and the inability of the Pegkeeper to respond rapidly enough to dampen upward pressure on the peg. This had knock-on effects that inadvertently caused liquidations in other Curve Lend markets.
This incident report will provide further clarity around the cause of the event, explain the reason for the Pegkeeper's unexpected behavior, and offer potential solutions and mitigations to improve crvUSD's resiliency to similar events in the future.
UwU Exploit (June 10)
UwU Exploiter: 0x841dDf093f5188989fA1524e7B893de64B421f47
crvUSD Borrower: 0x6F8C5692b00c2eBbd07e4FD80E332DfF3ab8E83c
Exploit tx: 0xb3f067618ce54bc26a960b660cfc28f9ea0315e2e9a1a855ede1508eb4017376
On June 10, an exploit in the UwU lending platform allowed an attacker to steal collateral held in the contracts, including a substantial amount of CRV. The UwU exploiter deployed the crvUSD Borrower contract linked above. In a series of transactions, the exploiter managed to manipulate the oracle used by the UwU markets to steal ~$19.4m of various collateral types held in the protocol. An overview of the exploit has been covered by rekt.news.
Of particular interest for Curve Lend is the tx labeled Exploit tx above. The tx at Block 20061322 (Time: Jun-10-2024 12:06:35 PM +UTC) shows that 23,617,586.248 CRV was used as collateral to borrow 8,123,821.056 crvUSD by the crvUSD Borrower contract. Instead of swapping the CRV in DEX pools (Total CRV exploited from UwU: 25,326,128.47), the exploiter chose to max borrow crvUSD for his CRV (23,617,586.248 CRV) because this way he can extract the most value possible for his CRV stack.
Shown below is the Curve Lend borrow operation within the Exploit tx, one of many operations that took place:
Source: BlockSec MetaSleuth
Exploiter CRV Liquidation (June 10)
The exploiter's borrow position, having been opened at the highest allowable LTV to maximize value extraction, was immediately at high risk of entering soft liquidation. Due to the relatively low on-chain liquidity for CRV and the abruptness of the event, there was also an increased risk of liquidation cascade. Heightened volatility would exacerbate the erosion of the position's health, making hard liquidation occur more quickly.
Note, however, that due to the soft-liquidation design, the worst scenario would be if the CRV price increased rapidly. In the LLAMMA soft liquidation algorithm, position healths erode by incentivizing arbitrageurs to swap between the collateral and debt token at a cost to the borrower. If CRV increased in price, the position would become fully exposed to CRV and simultaneously drop in health, potentially resulting in a hard liquidation of the full 23.6m CRV collateral amount. In actuality, the price declined, the soft liquidation mechanism began transitioning exposure to crvUSD, and finally, hard liquidation occurred.
Source: Twitter - CurveCap | Date: 6/10/2024
According to Michael (Curve Founder) in a Twitter post on the incident, the soft liquidation mechanism gave time for liquidators to prepare funds to OTC-liquidate the position. The system appeared to have worked as intended, with full liquidation of the exploiter's position and no accrual of bad debt to the Curve Lend market. Note also that Curve Lend markets are siloed, so even if there were bad debt accrued as a result of the event, it would only affect lenders to the single market pair.
Liquidation Cascade (June 12)
Although the UwU exploiter's position was successfully liquidated with no apparent negative impact on Curve Lend, it may have contributed to a liquidation cascade in the following days. As a result of the exploit event, there was 23.6m CRV unexpectedly introduced into circulation. This may have contributed to downward pressure on the CRV market price.
See below the sharp drop in the CRV/USD pair on June 12, bringing CRV down to a price of ~$0.20 before recovering somewhat.
Source: GeckoTerminal
Although the exploiter's position had been cleared two days prior, nearly 200m CRV continued to be supplied in the CRV-long Curve Lend market. This amounted to ~$70m worth of collateral at the time of the liquidation event.
Source: CRV Curve Lend Market - Curve Monitor | Date: 6/29/2024
At its peak, there was $55m of debt in the market, with the collateralization coming under threat on June 12 when a declining CRV price began a liquidation cascade that wiped out a major position in the market.
Source: CRV Curve Lend Market - Curve Monitor | Date: 6/29/2024
Lenders, presumably hoping to avert exposure to possible bad debt during the event, withdrew crvUSD and drove up borrow rates to the max allowable rate.
Source: Curve-Price API
The falling CRV price made the position vulnerable to liquidation and eventually, the position was liquidated. Shown below is a more focused look at the debt vs collateral value supplied to the market over time. Observe the phase where total debt and total assets are almost equal. Both the debt and collateral asset value decreased, which can be inferred as lenders taking their crvUSD out amid the crisis.
Source: Curve-Price API
Here is a more precise view of CRV price during the event. It shows the CRV-long market AMM price against the market EMA pool oracle price.
Source: Curve-Price API
crvUSD Upward Depeg
To carry out the liquidations, crvUSD demand increased substantially. This is because the borrower's debt was in crvUSD. In order to liquidate the positions, the liquidators have to purchase an amount of crvUSD almost equivalent to the user's debt to buy back the collateral (The collateral is discounted, but this can be discarded in the analysis). crvUSD can be purchased in Curve pools or alternative secondary markets. The supply of crvUSD should be regulated through the crvUSD PegKeepers, which deposit and withdraw crvUSD to specific Pegkeeper pools in an attempt to maintain the peg at the $1 target.
Analysis of crvUSD liquidity in the PegKeeper pools indicates that a large purchase of the borrower's collateralized debt position will have nonnegligible impact on the price of the stablecoin.
Source: Curve-Price API
What we expect to happen in such a scenario is:
the liquidators purchase crvUSD to buy back the borrower's collateral,
the price of crvUSD rises,
the PegKeeper contract deposits crvUSD in the associated liquidity pool to push the price downwards.
However, as we will show, the PegKeepers were prevented from providing by their associated Pegkeeper Regulator. Hence the price of crvUSD spiked dramatically during the liquidation cascade and remained above peg over many blocks.
Source: crvUSD Aggregator
Although Curve Lend is a siloed lending market, meaning the risks associated with the markets should be limited to lenders and borrowers in individual markets, this upward depeg of crvUSD affected users in markets other than the CRV-long market. This is because all Curve Lend markets denominate their price against crvUSD rather than USD. The spike in crvUSD price therefore is interpreted by the market as a reduction in the counterparty asset's price.
In some cases, such as the sUSDe-long market, there is a reasonable expectation that the counterparty asset (in this case a synthetic USD token staked in Ethena) will keep a relatively stable price, encouraging users to borrow at higher LTV ratios. The recently created sUSDe market was deployed to allow up to 35x leverage on sUSDe, which was advertised by the Curve Twitter account on June 9. Users may have been prepared for the risk that sUSDe may legitimately lose its peg due to restrictions on redemption imposed by its staking mechanic, but likely they were unprepared for the scenario where crvUSD depegs upward, causing the market to interpret their collateral as depegging to the downside. Users in the sUSDe market were inadvertently liquidated during this event. This demonstrated that, in certain scenarios, the siloed markets in Curve Lend, designed to mitigate risks to users, actually have generalized dependencies that may cause knock-on effects between siloed markets.
This scenario was not expected behavior, and the failure of the Pegkeeper to regulate the crvUSD supply was ultimately responsible for the depeg. The following section will describe the Pegkeeper design and why it did not perform as expected.
Pegkeeper V2 Performance
The Pegkeeper system was recently upgraded on June 3 to mitigate certain risks associated with the previous Pegkeeper implementation. The docs linked above go into detail about the new features, but there are two primary improvements to the Pegkeeper system:
Prevent Spam Attacks
Mitigate the risk of Pegkeeper stablecoin depegs
For the purpose of understanding the crvUSD upward depeg scenario, we focus on the prevention of spam attacks. Pegkeeper V1 had been vulnerable to spam attacks, whereby an attack could sandwich Pegkeeper updates by:
Flashloan and supply to Pegkeeper pool (balance the pool)
Call
Pegkeeper.update()to minimize the update amountwithdraw liquidity and repay the loan
The solution involves a check in the deviation between the pool spot price (get_p()) and the pool EMA oracle price (price_oracle()). This check would identify if flashloans are manipulating the pool price and cause the Pegkeeper update to revert in case the deviation exceeds a predefined threshold. Research on the historical deviations of the Pegkeeper pools suggested that a value of 0.0005 price deviation would be a generally acceptable value. The value would usually allow Pegkeeper to update while being small enough to prevent spam attacks.
Source: Pegkeeper V2 Research
According to research on this check, this attack is expected to be costly for the attacker and generate substantial fees for pool LPs, and is introduced to Pegkeeper V2 for additional resiliency.
For reference, here is a list of additional Pegkeeper V2 checks before it can provide and withdraw crvUSD to balance the prices:
Providing is not paused: This is checked using the is_killed method. If providing is paused, no crvUSD can be added to the pool.
Aggregated crvUSD price is higher than 1.0: The crvUSD price, obtained from the aggregator contract, must be above 1.0 (10**18). If the price is equal to or below 1.0, providing crvUSD is not allowed.
Action delay: Each Pegkeeper includes an
ACTION_DELAYconstant that limits how often it can be updated. The default value is 15 minutes.Pool Balance: crvUSD can only be supplied if the pool is <50% composed of crvUSD and can only be withdrawn if the pool is >50% composed of crvUSD. The update will reinforce the pool balance with 1/5 the quantity needed to perfectly balance the pool.
Price consistency check: The get_p (current AMM state price) and price_oracle (AMM EMA Price Oracle) must be within a specified deviation range (price_deviation). This is to prevent spam attacks by ensuring that the current price is not significantly deviating from the oracle price.
Depeg threshold check (for providing only): To ensure that the price of an asset has not depegged significantly, the current price is compared against a worst_price_threshold. This check ensures that prices across the pools with PegKeepers are within an acceptable range of deviation. If the price deviation is within the threshold, the PegKeeper is allowed to provide crvUSD.
Ratio limits: Individual Pegkeeper pools can only supply a portion of their overall debt cap until other Pegkeeper pools begin supplying to the pool. This is an additional mitigation against potential Pegkeeper stable depeg. See our Desmos graph for more detail on how ratio limits are calculated.
Pegkeeper Incident Analysis (June 12)
To analyze the Pegkeeper we need to check if it was active at the time of the liquidation cascade incident (with respect to the checks that need to be fulfilled before it can provide crvUSD).
For this analysis, we are analyzing Pegkeeper V2, as implemented on June 3. The following Pegkeeper contracts and corresponding Curve pools are listed below:
Description | PK Controller | Pool Address
crvUSD/USDT | 0xFF78468340EE322ed63C432BF74D817742b392Bf | 0x390f3595bCa2Df7d23783dFd126427CCeb997BF4
pyUSD/crvUSD | 0x68e31e1eDD641B13cAEAb1Ac1BE661B19CC021ca | 0x625E92624Bc2D88619ACCc1788365A69767f6200
crvUSD/USDC | 0x5B49b9adD1ecfe53E19cc2cFc8a33127cD6bA4C6 | 0x4DEcE678ceceb27446b35C672dC7d61F30bAD69E
Below we show an analysis of when the PKs were active and were providing and withdrawing around the incident. We highlight three pools overlaid with the crvUSD price spike to show the delay between the spike and successful supply by the Pegkeepers.
We will be concentrating on the three most liquid PegKeeper pools.
USDC/crvUSD
Source: Curve-price API | crvUSD Aggregator
USDT/crvUSD
Source: Curve-price API | crvUSD Aggregator
PYUSD/crvUSD
Source: Curve-price API | crvUSD Aggregator
The Pegkeepers were not providing at the time of the crvUSD price spike. This is not expected behavior. We would anticipate (as referenced by the Pegkeeper checks listed above) that as soon as the Pegkeepers register an aggregate value of crvUSD >1 and a deficiency of crvUSD in the target pool, they should immediately begin supplying crvUSD. That is, assuming anyone attempts to update the Pegkeeper (this is a permissionless call that shares profit with the caller).
Since the Pegkeeper was not active at a time when conditions certainly merited it, we provide an analysis of the checks that are made before providing. Following is the crvUSD price trend and price deviation check for the above-discussed Pegkeepers. When the deviation exceeds the +/- 0.0005 bands, Pegkeeper updates will revert.
We hereby analyze the deviation between the prices provided by get_p() and price_oracle in the three most liquid PegKeeper pools. We demonstrate that the deviation check does not function correctly for a considerable duration following the upward depeg event. This failure leads to PegKeepers refraining from supplying tokens in their respective pools.
USDC/crvUSD
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
USDT/crvUSD
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
PYUSD/crvUSD
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
Source: crvUSD Aggregator | Deviation Limit | get_p & price_oracle
A zoomed view around the depeg period reveals that in the three considered pools, the first update() was called ~525 blocks after the depeg began, around 45 minutes later. The deviation intermittently fell inside the tolerance range and the call was continuously made at regular intervals to continue supplying crvUSD to the pool, in order to bring its price back to peg.
Therefore, we observe that the deviation check prevented the PegKeepers from supplying crvUSD to their pools, thereby keeping the stablecoin above its peg. As we have explained, this behavior poses significant problems and carries consequences for the broader market. We propose several suggestions to address and eliminate this behavior from the system.
Recommended Mitigations
The liquidation cascade event in the CRV-long Curve Lend market on June 12 revealed a problematic feature of the new Pegkeeper V2 system, namely the deviation check between AMM spot price and EMA oracle price. The check was implemented to prevent spam attacks whereby an attacker could prevent proper action of the Pegkeeper by sandwiching the update() calls. Inadvertently, this check was responsible for preventing proper action by the Pegkeeper when a legitimate deviation in these values indicated a dire need for crvUSD liquidity.
The consequence of crvUSD depegging to the upside is potentially severe for the proper operation of the crvUSD system. For instance, crvUSD creates a proxy for its USD price by aggregating pool price data in its pairings with other stablecoins. An upward depeg can cause crvUSD to quote its collateral backing at a discount, potentially causing faulty liquidations in the crvUSD system. The risk is greater in the Curve Lend markets, where counterparty assets in the markets are priced against crvUSD directly, generally using EMA pool oracles in individual pools. Upward depegs are at risk of causing faulty depegs in Curve Lend markets, especially in markets that allow high leverage and involve correlated pairs (e.g. USDe/crvUSD) where users may assume more aggressive positions with the expectation of stable pricing.
There are several resolutions and available mitigation options to prevent an upward crvUSD depeg in the future:
Although the deviation check caused the Pegkeepers to revert in this instance, we now understand that it is possible to bypass this by doing the sandwich attack in reverse. One can flashloan crvUSD, supply to crvUSD pool within the deviation tolerance, call update, withdraw and repay crvUSD. A bot has been designed by the Curve researcher who developed Pegkeeper V2, and its code will be made publicly available
The deviation check was implemented in Pegkeeper V2 amid the recognition of a theoretical vulnerability for which it would improve Pegkeeper resiliency. It may not be necessary to depend on such a check, as the vector is costly to the attacker. In this particular scenario, an attacker would need to flashloan or otherwise acquire a large supply of crvUSD to spam the Pegkeepers. Given the low availability of flashloanable crvUSD, this is not currently a realistic vector. The deviation check could be turned off and reintroduced at a later time, when appropriate.
The ACTION_DELAY in the Pegkeepers is set to 15 minutes. Such a substantial delay may not be necessary. Reducing the value can further increase the cost of carrying out a spam attack. There may be a trade-off that reducing the value too much may prevent the emergency DAO from taking action if needed. This may be necessary if an undiscovered design flaw allows the Pegkeeper to supply excessive crvUSD into circulation. However, it may be acceptable to greatly reduce this value to 5 minutes without nullifying the emergency DAO's protective capacity.
Our view is that during times of market turmoil, when action from the Pegkeepers is most essential, they should be dependable and trivial to update. Therefore, the simplest solution is to modify parameters such that we ensure their dependable operation in all circumstances. This may not be an entirely straightforward solution, since within the current design it is likely that usability improvements are at odds with the usefulness of emergency backstops. We believe a suitable balance is to turn off the deviation check by setting its value to the max allowable (this can be modified in the future by DAO vote) and redeploy the Pegkeepers with an ACTION_DELAY of 5 minutes. We believe this value can adequately fulfill the original intention of the delay, still allow for emergency response if necessary, and mitigate the risk of spam attack enough to justify the full removal of the deviation check at this time.
Future Work
Spam Attack Cost Analysis
Future work involves estimating the cost implications of a spam attack by varying the ACTION_DELAY and spot from oracle price deviation parameters.
The discussion above may serve as an example where the spot price differs from the oracle price significantly due to a spontaneous price change in the market, and where we require immediate intervention from the PegKeeper contracts. Hence, one should investigate the possibility of completely removing the deviation parameter through the shrinkage of ACTION_DELAY, essentially augmenting the frequency of successfully calling the `update()` method.
In case where the call frequency has been changed, it should be verified it does not come at the cost of introducing new vulnerabilities.
Improved crvUSD risk simulations
Although not the focus of this incident report, the relatively poor on-chain liquidity profile of CRV precluded efficient liquidation of the CRV market position, leaving the market with a substantial amount of bad debt. This was notably a temporary condition, as the debt was able to be cleared as soon as CRV prices recovered sufficiently.
Curve Lend markets are permissionless and have no supply cap imposed by the DAO. The size of each individual market is largely limited by additional incentives allocated to lenders, typically by CRV gauge emissions. While it is impossible to enforce limits on market growth for collateral types that are at increased risk of liquidity related issues, it would be a great benefit for the DAO and for users participating in Curve Lend to have more accurate and readily available information about unsafe exposure levels.
To that end, LlamaRisk is building on previous work done by research teams like Curve Research, Xenophon Labs, and 0xreviews. These teams have developed simulation tools useful for assessing various risks within the context of the crvUSD protocol, and given the similarity of the Curve Lend codebase, can be useful also for lend markets. We will develop a risk portal dashboard that further increases transparency about risks related to individual Curve Lend markets and helps DAO members set appropriate gauge weighting.
Incident-specific Simulation
Additional research can be done on this specific market event for identifying further improvements to Curve Lend resiliency. For instance, we are interested in how the losses experienced from this scenario may have been lessened assuming infinite crvUSD liquidity or if the oracle was pegged to USD. Curve Lend markets experienced additional losses as a result of crvUSD depegging upward, although it has been observed that crvUSD mint markets (which are priced against an aggregated proxy for USD rather than crvUSD directly) experienced gains. It may be advisable to improve Curve Lend market resiliency by imposing similar limits on crvUSD oracle pricing.