Building Web3 Credit Scoring Capability (Part 1): Use of Funds
First..some context and intro
Atadia is a Web3 collective analytics business with a goal to reinvent credit and commerce with the power of data science and collective wisdom (read our update roadmap here).
One of the most important tasks on this roadmap is to build a Web3 credit scoring capability. Credit is part of PFPscore, a trust & reputation system we’re building to support the Web3 economy. Having a functional Web3 credit risk capability is essential to expand existing use-cases of DeFi beyond trustless ones.
Building Web3 Credit Scoring Capability entails the following:
- Machinery: Atadia has an automated engine in the back that can accurately estimate credit risk of any anon wallet owners
- Human Capital: Team Atadia, AtaDAO members, our DeFi partners, and Web3 supporters all get a better understanding of credit risk in the Web3 context
Attaining #1 is a significant challenge even with IRL alternative credit. Doing this well on Web3 is unprecedented, almost unimaginable.
This article is Part 1 of “Building Web3 Credit Scoring Capability” series.
Writing these serves as our effort towards capability #2, which in turn will help Atadia do #1 better in the long run + helping Web3 credit flourish faster and safer with more knowledge about risks going around.
❤️ If you’re a holder and are reading this, know that “the secret sauce” stays within Atadia 😉 and know that we’re so early it is important to share our experience and learnings so that the ecosystem grows as a whole.
*If you’re curious to learn more about our approach to credit scoring using Big Data, AI, and community involvement feel free to check “Knowledge Base” Channel or AtaDAO Announcement Channel in our Discord.
The Atadia Lending Lab
There are many ways to build a credit scoring engine.
One of the cleanest ways is to extend capital in an unsecured way and build a statistical model around the loan performance results.
You’ll learn soon that even with this “clean” set up that Atadia has complete control of, it’s still quite a mental challenge to try to solve and attain this capability.
Back in April, we launched 🧪 The Atadia Lending Lab 🧪.
It was likely the very first experimental uncollateralized lending service on Solana (maybe even beyond Solana…correct us if we’re wrong).
Small loans of 1–4 SOL were dispatched to borrowers as part of a data collection experiment to kick start model development, and they were asked to repay in 1 week with a 5% convenience fee (quite high in hindsight 😂 but it was mainly for external validity purposes).
Users connected their wallets on our website, applied for loan packages they would like, we crunched some numbers and returned underwriting decisions.
Note that the underwriting decisions were made with an experimental design in mind. They were not entirely based on our pre-mint model.
Borrowers can also check loan status in our Discord with a custom-made bot.
If you want more info, this thread does a great job at summarizing the offer.
We also sent 4 variations of “Debt Reminder NFTs” (ominous pictures of Detha, Atadia’s authoritative ruler) to wallets near due date or when they were past due.
As of today, The Atadia Lending Lab experiment has about 750 unique borrowers with ~1k loan applications accounting for over 2000 SOL in total.
3 Credit Risk Tasks for our Analytics Team
Combining data from The Atadia Lending Lab with external lending data by our partners (who mainly do collateralized lending), our analytics team have been busy with 3 major tasks:
- Credit Scoring Model Development — this mainly includes generating hundreds of features (observed behaviors about someone), cutting them down to the best, relevant ones, and productionizing these models
- Web of Wallets Algorithm Development — probabilistically solving “If A owns wallet X, what other wallets does A own?”
- Exploratory Analysis & Deep Dive — this is both for internal use and for sharing like in this article
Progress so far:
While it is too early to be confident and too easy to build a model that overfits the initial set of data, Task 1 yielded around AUC=80, an ok-ish predictive performance for first try. Quite ok at detecting the two extremes. Not so great in between.
This is with cross-validations and only on 1 wallet = 1 user sample. Theoretically, we should be able to attain higher performance as we expand to “wallets connected” or “all probabilistically webbed wallets.” But you will soon learn that credit risk on Web3 ain’t easy.
What we’re more content with is the fact that we’ve built our backend code to generate an arsenal of 400+ features. We aim to improve the architecture in the months to come so we can pin these against any new lending data in the future more swiftly.
Task 2 is a strange one. There are many methods to solve it and one of the first methods we tried worked almost 80% of the time. However, it’s virtually impossible to be 100% confident about it for all wallets on a blockchain.
In this article, we relay some initial insights from Task 3.
More specifically, what happened AFTER we disbursed the loans.
In addition to learning what pre-lending behaviors predict default status (Task 1), post-lending behavior is important for our team to grasp both from the business and the analytics angles.
Note1: This preliminary analysis focuses on The Atadia Lending Lab borrowers only. It does not include post-lending insights generated from partners’ data.
Note2: This preliminary analysis does not take into account the probabilistically connected wallets from Task 2, just the wallets that borrowers actually connected to the Atadia Lending Lab themselves. Some folks chose to connect 1 or more wallets. Therefore, it is entirely possible that the insights you see in this article can still be half-truths. We decided to share anyway just for transparency and we’ll iteratively readjust these in the future as we all learn more.
For you statistics-minded folks, the sample in this article is likely biased by confounders or colliders that determine the decision to connect how many wallets (and which) to our website.
Lending Take-ups & Outcomes
The word cloud chart below showing the top 30 NFTs our borrowers hold demonstrates the eventual diversity of our final group of borrowers. Some collections that they hold are Famous Fox Federation, the Stoned Frogs (our shilling there totally went viral), SC, Okay Bears, just to name a few.
For this experiment, we define “Default” status as non-repayment of over 2 weeks after the due date. For simplicity’s sake, we bundled non-payments and late payments into the same default category for this analysis.
In April, the Lending Lab’s default rates fluctuated between 7–12%, which seemed unusually low given the unsecured nature of the loans in this “trustless environment.”
To ensure our eventual model’s ability to detect risky borrowers as well as low-risk ones, we decided to strategically expand our reach to other parts of the risk curve.
In other words, if the model’s never seen a variety of risks, it won’t be able to perform for real. No credit risk model is good if it can only perform in a tiny section of the risk curve.
We did just that via marketing partnerships with pre-mint projects + being more lenient with underwriting.
The Lending Lab was shilled a few days before mint to whitelisters in various projects such as Vault-X, The Stoned Frogs, Solful, etc. The initial goal was to get exposure to the right audience — potential minters who may be looking for liquidity. Eventually this became a borrower targeting tool.
As anticipated, one-week default rates gradually climbed up to a grand total of 36%.
What do most borrowers do with their loans?
They send the money out to another wallet!
This activity accounts for 52.5% among non-defaulters and 81% among defaulters. The remaining subjects used the loan immediately from the wallet that they received the funds either to buy NFTs from a secondary market or to mint.
Clearly, a vast majority of borrowers, defaulters or otherwise, interacted with both our lending lab and other lending protocols using their 🔥 burner wallets.
To gain a better understanding of what people actually do with the money, we therefore need to dive deeper to their “next” wallets, to the next level of connection…
Ok before we go there, you might be wondering HOW we determine if a certain activity like a SOL transfer or a purchase on ME was indeed the borrower’s intended use of the loan.
The answer, unfortunately, is we don’t.
Unlike NFTs or the Handshake tokens where each person holds only a single unit and each transaction of said token is definitely the owner’s intended use, SOL is plentiful in your wealthy wallets, and we are cornered into some degree of detective work.
Our underlying assumption here is that your next SOL transaction after receiving the loan was what you indeed planned to do with the loan. It’s probably not perfect, but believable if people behave like IRL.
The Need for Recursive Data Digging (and Web of Wallet Algo)
Using the same logic, we tracked down the actions that took place in the connecting wallets to which borrowers transferred their loans.
Given the limitation of time and resources, we went only as far as 3 layers deep for this article (so a total of 4 wallets in each thread of SOL transactions).
Loan usage is defined as the “first” non-SOL Transfer transactions recorded within 4 levels of connected wallets immediately after receiving the loan. If there were no non-SOL Transfer transaction observed within these 4 wallets, we marked them as “SOL Transfer” usage type for now (bottom row).
The chart above conveys two useful points:
- Most defaulters definitely have more than 4 wallets— at their 4th wallet, half of them are still transferring SOL to the 5th. This is useful behavioral pattern for Task 1 and Task 2. It also means that if we need to dig, we need to dig real deep and that’s ok 👌
- Loans predominantly were used for mints and NFT purchases from the secondary market. Non-defaulters appear more open to buying from the secondary market than other uses.
We can’t wait to revisit these results again when our Web-of-Wallet algo is ready for real.
Revisiting 7-day Tenure Loan
Back when we internally discussed the appropriate loan amount and durations for the lending lab, the team debated hard about potential uses for the loan.
We even ran a contest on Twitter a few weeks into running the Lending Lab to get a better feel of what was going on. The most obvious use for a loan with a tenure this short was for quick flips, either through mints or through secondary buys — the Twitter contest supports this belief, so that was the overarching hypothesis.
Digging beyond “first immediate” usage, we expanded our analysis across 7 days of the tenure and found 2 interesting types of behaviors among our subjects.
First, the majority of defaulters seem to be on the edge of their seats waiting for the loan to arrive 😂. Makes sense if you’re out there to steal right?
Once it did, they were really quick to transfer that loan out to their other wallets (see chart below on the right). The more creditworthy folks or “normal” folks on the other hand (LHS) are slower and use their wallets for a much more diverse range of activities.
Second, contrary to our initial hypothesis and anecdotal evidence from chatters 😂, the average borrower does not seem to exhibit the behavior of a flipper.
Most received and spent some SOL/SPL, but did not mint, buy NFTs from the secondary market, nor did they sell NFTs within the loan terms.
Note that this does not mean that nobody is minting at all or is making some quick bucks within 7 days before repaying the loan, just that “the average borrower” does not.
The figure below reaffirms this finding — on average, borrowers bought NFTs & HODL, mint & HODL and bought & listed less than 1 time in 7 days.
None successfully finished the cycle of buy & sell NFTs within the duration of the loan, at least from the perspective of the wallets connected to us.
It suggests a possibility that at least half the people do not view our service as a money making machine, but rather a convenience loan service or a tool to build their credit scores.
What’s next?
So far…here’s what we learned
- It is certainly possible to do credit risk on Web3 or even control unsecured risk, but it really isn’t a walk in the park both on the data science side and on the business side.
- Defaulters and non-defaulters behave differently in many dimensions from the diversity of their activities, how actively they transact, the sort of activities they do day-to-day to the number of wallets they own. We’re just scratching the surface.
- There is a need to dig really deep both for pre-lending and post-lending, possibly with the help from Web of Wallet algorithm (Task 2)
- More research is needed to figure out why people paid back (anecdotal evidence so far suggested that a) folks actually want to start anew and build credit here 🤝 and b) folks fear being permabanned from ecosystem-wide services like being part of a DAO or getting WL privileges.
There’s still so much we don’t know and we cannot wait to do more analysis + share what we learn in future “Building Web3 Credit Scoring Capability” series articles.
👀👀 The Atadia Lending Lab is currently closed but will soon reopen for the next batch so keep an eye out!
See you all next time!
🥂🥂
