Lilypad Network
Lilybit rewards emissions issue #14
Replies: 9 comments 49 replies
-
|
A bug in the rewards calculation has been identified and a fix is in progress. After looking through the rewards data from the last week, we believe the issue has been identified, but will continue to monitor. We will keep the community updated with status and next steps. In addition, our team is looking at the ideal method to normalize points moving forward given the last few days of large numbers of Lilybits sent out. |
Beta Was this translation helpful? Give feedback.
-
|
Really happy to hear this. Tyvm. |
Beta Was this translation helpful? Give feedback.
-
|
This RP got 1838 LP on the 12/11 after missing 1 hour of POW, on the 13/11 it received 0 LP even though it didn't miss any POW for the 24 hour period. 1x H100 SXM5 80 GB, 32 Intel Xeon Platinum 8470 vCPUs, 32 GB RAM, 120 GB, Private RPC 0xAce37E2fDFE20E88CECa68B092c91bB579112126 RP LOGS : Will attach, not sure how u want these as can only attatch 25MB worth, its the first 25MB worth of the logs too, any extnesive logs in comment box seem to cook the webpage and I cant post. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for the context! very helpful while our team investigates. |
Beta Was this translation helpful? Give feedback.
-
|
FYI another day of 0. Have this across 70% of my nodes. Different specs, different providers. |
Beta Was this translation helpful? Give feedback.
-
|
Please submit all details via a ticket issue. No investigation will occur now without full details now in order to reduce overhead on support staff. Thanks |
Beta Was this translation helpful? Give feedback.
-
In here? Or open a ticket in discord? |
Beta Was this translation helpful? Give feedback.
-
|
would ask to open a new github discussion here. We are tracking one sprint of work with the current discussion and will use the new discussion to track other work. |
Beta Was this translation helpful? Give feedback.
-
Update:While analyzing network metadata, we identified certain patterns that suggest a potential discrepancy in resource allocation. Our investigation indicates that some Resource Providers may be registering multiple wallets on a single GPU, which is not in line with our intended usage guidelines and is detrimental to the network.
We have seen this significantly contribute to the increased daily points emissions once the new rewards calculation went live. Our team is taking steps to address this issue and ensure fair participation within the Lilypad ecosystem. As stated in our docs: "In order to run jobs successfully, Lilypad RPs must ensure all resources on the server are available to the RP." Work in progressA fix is in progress and we will announce more details as work completes. The large number of RPs contacting the rewards system seems to have caused some of our rewards system scaling issues (we have identified many cases of nodes running many RPs on a single GPU) . As this work completes, we believe the rewards system will function as expected. Once the system is updated, our team will monitor the rewards data to evaluate if this fix fully resolves concerns brought up with the rewards system and communicate with the community (including how we intend to normalize the points that went out this week). Its likely we will use the slashing mechanism to penalize RPs running many RP nodes on one GPU (on all wallets but one) to work to normalize this situation. As mentioned in the original Lilybits rewards blog, we will continue to upgrade the rewards mechanism (as fairly as possible) to match the utility needed for Mainnet. In order for the Lilypad network to be successful, larger GPUs will be required and will naturally earn more rewards. The next rewards system upgrade will likely take the form of a job based rewards system (run jobs across the network consistently to prove RP uptime, and RP has full access to resources). Please contribute data to this Github discussion or open a new a Github discussion if needed. All related info helps our team investigate and find solutions. Find a guide here for submitting issues for support. Our team appreciates your understanding and cooperation as we work to maintain a balanced and equitable platform for all users! 🚀🚀 |
Beta Was this translation helpful? Give feedback.
-
|
It is not incumbent on us. This is incorrect. Not being able to run jobs due to overallocation is not in the spirit of the program and im extremely confident you knew this was a risk when you made the choice to do this. Changes should always be expected in a testnet and we always stated - repeatedly- that those setups not owned were a risk. It is very common for engineering timelines to extend and extremely common that programs change in testing phases to suit the needs of the overall health of the network. These cannot be considered unknown risk factors in decision making. The decision is final here. |
Beta Was this translation helpful? Give feedback.
-
|
Will the minimum specs remain the same through mainnet? |
Beta Was this translation helpful? Give feedback.
-
We asked multiple times if this setup was acceptable, and you confirmed that it was. Reversing your stance now and penalizing the community reflects poorly on the project's leadership, especially given the prior assurances documented above via a screenshot.
Please provide the full mailing address for your company for the purpose of delivering an official letter. |
Beta Was this translation helpful? Give feedback.
-
|
Multiple wallets to a gpu that couldn't run jobs was not allow listed explicitly. Nor was this questioned multiple times Regardless, our decision is final. The health of the network is the paramount goal for us and should be for all RPs. Without this, there are no rewards of value for anyone. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Ally, Thank you for the update. I'm not surprised that this was the cause of the spike but at the same time, this directly contradicts our conversation on Discord on November 5th: I can understand why it may not be ideal to overallocate a GPU, but at the same time, proof of work systems are supposed to deal with this situation and require the RP to prove the hashpower available on their rig. And that seemed to be the case prior to the last reward calculation change. You cannot blame the community for working to optimize within the parameters of the system, and as others have mentioned as well, you shouldn't penalize folks for consolidating to fewer physical GPUs as a means to keep nodes active without being slashed. If running multiple nodes per GPU is against the rules, it needs to be documented in the rules. If that is a rule going forward, so be it, but you should give existing RPs a chance to adjust without being unfairly punitive. My suggestion would be to either switch (back?) to a POW system that goes off the true hashpower of the entire rig (which is what I thought was the case to begin with). That way, an overallocated GPU would report a lower hashrate per node as several POWs are happening at the same time. Alternatively, end the slashing program for nodes prior to this rule clarification so folks can safely turn off nodes to get back to a 1:1 ratio moving forward. In either case, I am totally supportive of adjusting the rewards for the past few days to bring emissions back to where they should have been before the last change. That makes sense and is most fair to the earlier node runners who have been diluted recently. I get your frustration with this discovery, but please recognize that it is incumbent on the project to provide clear rules for RPs to follow. This situation could have been avoided by a single line in the incentivized testnet documentation that states "only one node/wallet should be allocated per GPU". Lesson learned I hope, but please don't place the blame on the RPs for not following a rule that wasn't documented. At the end of the day, the RPs power the network and are volunteering a lot of time and resources to help you achieve your vision. Onwards and upwards. EDIT: I can imagine the impacts of realizing that a lot of the nodes and hashpower you thought was there might not actually be there, and that creates all kinds of problems for the project. But this is a chance to get things right with the community that supports you. The above proposals will help with that. But punitive action is going to push away big supporters and further reduce the resources and hashpower available. In my specific case, I'd be happy to bring on more GPUs and get to that 1:1 ratio but I need notice and some assurances that previous rewards would not be slashed unfairly. I still think the best solution is going off available hashpower as this seems like a more sustainable, realistic, and fair approach to handling the dynamic nature of consumer-provided hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Overallocating has caused issues and is not in the spirit of the network when these gpus could not run jobs. This was made clear from the very first blog. While I agreed it wasn't cheating, it is also not of benefit to the network. Should we have designed it differently? In hindsight, yes, however that is what we are correcting now and why it is a testnet. I think everyone that went down this path knew they should have checked and knew it wasn't right to do nor in the spirit of what the network was aiming to build. The only checking that occurred was when we changed. Not prior. To clarify on your discord image also - I did not think you were using multiple wallets with one gpu. Just optimising the output of the GPU with multiple processors or I would have responded differently. Regardless, this was only mentioned AFTER we changed the mechanism, not before it was done. It's now come to light that yes, that wasn't good for the health of the network, it's being corrected and people are unhappy about being called out on trying to extract more out of the system instead of contributing to building the value of the system fairly and in keeping with the aims of the network intent. The RPs should be powering the network. That's correct, but they are actually causing it to be unusable instead because they are not able to run jobs. This is not useful work and every RP would be very aware of that not being in the spirit of what rewards were provided for. In fact, I'd guess many conversations were had around this topic behind the scenes. It cannot be argued, as you are trying to do, that this wasn't clear. This was not operating in good faith. It is laid out in the original blog post. Now, we are doing the right thing by every provider that did the right thing by the network and more importantly by the network itself by correcting this. We are here, to get things right with the network. And be a sustainable business of utility and value which rewards according to this. That is our primary goal and should be the primary goal of network participants. We are here to build real value. We are here to reward real value. We are here to be an alternative to the extractive practices of centralised systems and instead distribute value fairly to those providing it across all layers of the stack. This is the community we are building. |
Beta Was this translation helpful? Give feedback.
-
|
Fair enough, sounds like a lot of misunderstanding on both sides. But we're here now, and the end of the day the RPs still provided you hardware free of charge. How you choose to respond to this situation now is going to influence the response from the RPs. I'm glad the rules have been clarified so we can all move on, but we will need to see the details of how the situation will be resolved retroactively. I wouldn't say I was acting in poor faith by calling to your attention how rewards jumped when the last change was made. I, like many others, assumed that a POW system meant your entire system is being scored on the work it is able to do. Optimizing within those parameters when your GPU is underutilized (and the data backed this up... I never saw the 4090s go above 260 watts) just makes sense from a provider standpoint. I was specifically concerned about this very thing. I'm a new RP and have only been running for a few weeks. Now I'm faced with a situation of having to find homes for many nodes when GPU hardware is scarce and expensive. So I hope whatever action you take accounts for those realities, and gives paths for folks to make it right, vs labeling them as bad actors and punishing them for optimizing within the confines of the system. |
Beta Was this translation helpful? Give feedback.
-
|
We specifically have said many many times that 3rd party gpus are not a good idea. |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate the clarity that has finally been provided around this issue. And I understand team's point of view here. I have joined around the time where others asked a question about underutilisation of resources we provide. My intent has been to support the network by ensuring that hardware is fully utilised and provides real value. According to messages posted by you, Ally on Discord - it was not considered cheating so I understood that maximising my GPU usage is encouraged, especially under a PoW system where rewards should be based on output. Like many others, I assumed that since a single GPU could handle multiple containers without any indication that this was disallowed, it was within the network’s guidelines. When the hardware is consistently underutilised, it makes a logical sense to experiment with optimising its capacity to maximise contributions. Given that no explicit rules were set against this and that team members indicated such optimisation was acceptable, I believe that most actually acted in good faith and I really hope that any retroactive measures take into account this initial lack of clarity on this matter. I hope that rather than penalising providers who operated within an unclear framework, you will figure out a way that doesn't slash entirety of previously earned rewards and allows transition into aligning our configs with Lilypad network's intent. I think we all invested significant resources and time into supporting it. |
Beta Was this translation helpful? Give feedback.
-
Could you elaborate on the very first part of this message. A bit confused. It was okay for multi wallet gpu this week only? I thought that's exactly what wasn't okay and that's why emissions are so high? |
Beta Was this translation helpful? Give feedback.
-
|
It was not. As I made clear earlier. |
Beta Was this translation helpful? Give feedback.
-
|
But what actually is the decision to deal with this situation? I get it's 1:1 going forward but what does that mean for existing nodes? We'll need absolute clarity on this. It would be in the network's best interest to provide a path that allows RPs to maintain what they've earned by transitioning to new hardware to maintain a 1:1 GPU to wallet relationship. Punitive action such as slashing all but one wallet will only harm the network by discouraging folks from spinning up additional hardware to maintain rewards. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. With all due respect you really haven't made much clear at all. |
Beta Was this translation helpful? Give feedback.
-
|
It is 1:* wallet to gpu. We will be implementing this and yes slashing will be on. People that were using multi-wallet per gpu will be slashed unless there is a gpu on each wallet. Correct. The only question asked on this was by you a week ago Isaac. If we locked any points, it would only be from that misunderstanding and no earlier |
Beta Was this translation helpful? Give feedback.
-
|
Tldr
|
Beta Was this translation helpful? Give feedback.
-
Sounds great for grace period. |
Beta Was this translation helpful? Give feedback.
-
|
Ally, I urge you to reconsider your stance on this issue, as accountability is crucial, especially in a testnet environment where design flaws are expected. Projects that penalize miners for optimizing their setups, rather than addressing their own incomplete documentation or flawed code, often face significant backlash. They struggle to attract venture capital and additional resource providers, and ultimately, they tend to fail. Although I'm not personally affected, I've observed this pattern consistently across various projects. Instead of penalizing miners for seeking optimal configurations, it would be more constructive to take responsibility for the protocol's shortcomings, address issues with resource providers, and move forward collaboratively. This approach is essential for building a strong, cooperative community and avoiding unnecessary conflict and animosity. |
Beta Was this translation helpful? Give feedback.
-
|
We have taken responsibility and are changing the shortcomings as one does in a testing environment, however, multi-wallet per gpu that couldnt run system jobs wasn't optimising, it was cheating the system. It was clear from the beginning that would not be tolerated. It also is not good for an overall network... ie what venture capitalists want to see. As discussed earlier in this thread, the addition of retail gpus is only part of the solution also and I have no doubt gpus will go where they are paid. A sustainable network can provide that, as well as a stable economic model for providers. One that can't run, cannot. |
Beta Was this translation helpful? Give feedback.
-
|
While I understand your point of view, it’s important to note that if the protocol was rewarding those actions, then technically it wasn’t "cheating." Nothing in the documentation explicitly stated that using multiple wallets per GPU wasn’t allowed. If the system was designed to accept those actions, then it’s more of a protocol oversight rather than intentional misuse. |
Beta Was this translation helpful? Give feedback.
-
|
We reserved the right to make and correct those mistakes - this is what a testnet is for. If you had discovered this, it would have been prudent for network participants wanting a strong, successful project to directly bring this to our attention. This was not the case, though I know as a certainty that discussions on how to leverage for individual (not network) advantage happened behind closed doors. So, yes. It is cheating the spirit of what the network was designed to do. Let's not pretend RPs didn't have an understanding that was what they were doing. If there was good faith the question would have arose much earlier. |
Beta Was this translation helpful? Give feedback.
-
UpdateAn update to the Lilybits rewards system is now in testing/review. We'll push the update as soon as possible and will announce to the community when we have done so. Only one wallet (RP) will be eligible to earn rewards per GPU. For RPs running multiple wallets on one GPU, the wallet with the most points will be matched in our system with the GPU and allowed to earn points while the others will not earn points (unless they are used on another GPU). Slashing will continue to be turned off temporarily (as a grace period) until we are certain the rewards system is working as expected and to allow time for RPs to adjust.
The rewards system will then be monitored. We are aware of the large number of Lilybits provided this week and this fix will in part help to resolve this. Our team will look at the rewards emissions data to see the impact of the update. We will then have enough info to decide on next steps to ensure the system is fair given the rate of Lilybit emissions over the last week vs in the previous months of IncentiveNet. More details on this will be announced. We also believe this fix will resolve issues with RPs getting 0 rewards and will monitor this issue. For RPs not getting points and running PoWs actively (with 1 GPU to one wallet), we are looking into a retroactive backfill. Thank you for the patience while we carefully update the rewards system. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the update Alex. Appreciated |
Beta Was this translation helpful? Give feedback.
-
|
As this work is finishing up, we have locked this discussion and will close it upon work completion. For support troubleshooting other related issues, please open a new discussion and provide as much detail as possible. Check out the existing discussions to see if theres an ongoing issue affecting your RP. |
Beta Was this translation helpful? Give feedback.
-
Update: Implemented initial fix for Rewards SystemOur team has implemented a fix that will address the large number of daily Lilybit emissions. Changes take effect today and will reflect in the total daily Lilybits sent out going forward. Discord announcement here. Only one wallet (RP) will be eligible to earn rewards per GPU. We'll communicate next steps on rewards emissions concerns once enough data has been collected on the results of this work. https://github.com/orgs/Lilypad-Tech/discussions/14#discussioncomment-11261561 To report additional RP issues, open a new github discussion by filling out this template or check out any existing discussions for your issue. For troubleshooting help, open a ticket in the i-need-help Lilypad Discord channel. |
Beta Was this translation helpful? Give feedback.
-
Describe the problem
Identify the cause of total Lilybit daily rewards increasing by 2x - 3x when compared to the daily rewards earned on the old (hashrate based) rewards calculation.
RP hardware and software info
N/A
Eth address (Wallet address)
N/A
RP logs
N/A
Beta Was this translation helpful? Give feedback.
All reactions