Codespaces storage chewed by itself

[mountainash]

Select Topic Area

Bug

Body

Seems to me like a possible bug...

Each day, for the last 3 days, I've been emailed an alert about my "GitHub Codespaces storage usage" using an increasing % of included services, yet in this same time all my codespaces have been stopped (see screenshots).

How is it possible that the available storage space is being increasingly chewed by spaces which aren't running? Seems like a possible bug with the reporting or are these spaces working silently in the background?

[kblake]

Same with me. I am confused. This prompt to "update spending limit" has come from somewhere recently. I don't even know what I'm doing to use up Codespaces storage!

[treeder]

Ditto, keep getting emails that I'm using more and more storage, but i'm not using more and more codespaces.

[robhawkes]

Same issue here, all 15GB vaporised within days with little to no Codespaces use. There's no way to know which Codespaces is taking up the space so it's not possible to know how to fix the issue, so there's no chance that I'm removing the spend limit.

I've just submitted a support ticket so hope to hear back soon.

[mountainash]

This ⬆️ should be a feature; if storage size is a plan limit, there should be a simple way to see the size of Spaces (along with the RAM and CPU allowance)

[mountainash]

@robhawkes did you ever hear back from Support? This is still an issue for me (nearly 1 year later).

[wuwuwu]

Same here, 15GB destroyed...

[samkio]

Same issue here. It would be nice to see where the storage is going towards? Yesterday I got the 90% warning and just now I got the 100% despite not using a single codespace...

And now I can't access the codespace before it will be deleted. 😢

[mountainash]

Seems it's not just me then... does anyone at @github-staff read these boards? Can they confirm that this is an issue?

[samkio]

I found this related discussion https://github.com/orgs/community/discussions/38920#discussioncomment-4140848 which @craiglpeters has commented on and sheds some light on this. It appears storage counts regardless if the codespace is active or not.

[mountainash]

I get that space "counts" while the container is not active (due to persistent storage) - but this doesn't explain how the amount stored can increase while containers are not in use. The amount disk space used should only fluctuate while a container is active. When the containers are suspended/stopped, then the amount of disc space should remain unchanged until a container is run again.

[samkio]

It's because it is time based. If your codespace has 1GB of used storage but it only exists for half a month then you would consume 0.5GB of your quota.

Hence our codespaces must have been >15GB and as time goes on it contributes to the quota and thus gives the impression of disk use increasing when in fact it is the same.

[samkio]

A better example is a codespace with 30GB of used storage. At the half a month time we will reach the 15GB quota regardless of what state the codespace is in.

[ssbssa]

You should read how the storage is calculated here:

Storage is billed for all of your existing codespaces, regardless of whether they are active or inactive with the exception of blocked usage due to exhausted included usage quota or reaching your spending limit. The storage billing for a codespace ends when it is deleted.

and:

Your GB-month storage is calculated as follows. Once every hour, the storage used by all of your currently active and stopped codespaces is assessed. This figure is then divided by the number of hours in the current billing month: total storage size / hours this month. The result is added to the running total for codespace storage for the month.

[martinlicht]

That basically means that any idle space, even if absolutely nothing happens in any of your repos, contributes to an ever-increasing total

If that's taken as written in your quote, then every GB leads to about 7.1 GB/hours a month.

Which is quite unintuitive and absurd. For example, that would mean that usage at the end of the billing cycle weighs in less than at the beginning. When reading further the example in the documentation, it seems they compute the running costs differently.

[robhawkes]

This has been resolved for me now and my usage allowance has been reset, meaning I've been able to go in to each Codespace to check / delete the ones that I no longer need.

This was the email I received, as well a reply to the support ticket that basically said "check the email you just received":

Codespaces Customer,

Since we launched Codespaces for All including free usage and ended the beta for individuals, we have noticed that many of you have old codespaces that were created prior to the availability of retention periods on 22 Jun, 2022 and thus do not have a retention period applied.

In order to help you manage your storage consumption below the free usage allotment, on 21 Nov, 2022 we will apply a retention period to your older codespaces with an expiration of 5 Dec, 2022. If you do nothing, those old codespaces will be deleted on 5 Dec. When you use a codespace, the retention period is restarted, so you don't have to worry about actively used codespaces being deleted.

Codespaces consume storage while they exist - for more information see "About billing for GitHub Codespaces." Because these older repositories didn't have a retention policy, even if you haven't used it it a long time, your storage is being consumed by these old codespaces. You can see a list of your codespaces at https://github.com/codespaces - a link you can now find quickly from the top of most GitHub.com pages.

Because Codespaces are ephemeral and can be quickly spun up for your work, you can safely delete them once you commit the changes.

Thank you for your use of Codespaces.

Best Regards,
Codespaces Team

[mateeni-dev]

The disk space used by all of your codespaces and prebuilds is assessed once an hour and your current GB-month usage is recalculated. Therefore, while you have codespaces and prebuilds, your GB-month usage will increase throughout the month

ok, but why?
how does this even work?
1GB now should be 1GB 30 days later, in terms of their actual storage used. or what am i missing here?

[louh]

Yeah, this is super weird. I just got an email saying that I'd used 75% of my Codespaces storage, and I don't even use Codespaces. I was tearing my hair out trying to find if I'd changed anything recently that would cause it, or if something else (e.g. GitHub Actions, or Dependabot, or some other service) was actually using Codespaces under the hood. Or if someone had figured out some security loophole and was able to use my storage space for nefarious purposes. Finally came here, and things started clicking.

I wish the list of Codespaces was easier to find. Once I went there, I found a couple of very old spaces that I had started back when I was first giving the feature a try -- these were a year or two old. I have no idea how they're using 1.5GB a day, but ....

Anyway, I deleted those spaces, here's hoping the usage report clears up.

[tristan-f-r]

I had this issue on the first few weeks of release, but the whole "20gb" thing needs to be fixed. It makes codespaces virtually unusable compared to competitors like gitpod, even though github already gives you a storage limit of 50gb in repositories. GitHub either needs to fix their storage usage benchmarking or upgrade the amount of provided free storage.

[ArielMAJ]

This is happening to me too. I have deleted all my codespaces a few months ago and haven't created new ones anymore and this keeps happening. Does anyone have a tldr easy solution on this (or any solution at all)?

[ArielMAJ]

See https://github.com/orgs/community/discussions/39133#discussioncomment-6004433

[garethpaul]

+1 This is happening to me, I'm being billed for Codespaces storage but have deleted all my codespaces instances. I've read through the docs but can't figure out how to delete the storage that I am being billed for.

Any help here would be greatly appreciated? cc:: @tanmayeekamath

[matthewoden]

I had an issue with this recently as well. I went to billing, and ran a usage report. Looking at the result, I think I'm a little confused.

For others looking to do the same -- look at Settings > Billing > Generate Activity Report. You'll see repos with activity and amounts.

There seems to be a calculation issue. I went from 75% to 100% of usage overnight. My only activity during this last billing cycle is prebuild storage for one repo, which is using 3.7056 per gb-month. It should never have gotten anywhere near full.

The prebuild was set to only run on push, and there were only two action runs that generated prebuilds, both from months ago. This month is the only month I've received alerts. I would imagine something is broken.

[ArielMAJ]

Tyvm I believe this info is what I needed. I generated my activity reports as well and found which repository was using all my storage. It was a similar thing, a prebuild template that run on PRs and pushes to main. Every time it ran it chewed up almost 4gb it seems. When accessing my settings it wouldn't show up, but by accessing the correct repo I could find and delete it. Hopefully this will stop happening now.

tldr solution:

1. https://github.com/ > Settings > Billing > Generate Activity Report > Download Report in your email > Open CSV > Find the repositories that are consuming your space.
2. Access those repositories' codespaces page: https://github.com/<YourUserName>/<YourRepoName>/settings/codespaces
3. Delete prebuilds & codespaces;

[TamerTawfik]

Thanks, that is exactly what I needed.

[JavierG123]

Genius!

[voljumet]

I have never used codespaces, and now it says that I've used it all 😆
"You've used 100% of included services for the account"

[ArielMAJ]

@voljumet See if this helps https://github.com/orgs/community/discussions/39133#discussioncomment-6004433

[Solido]

Idem here. Got 0 spaces and keep moving to limits.
This thread is not even read by github team :D

[ArielMAJ]

@Solido See if this helps https://github.com/orgs/community/discussions/39133#discussioncomment-6004433

[Solido]

Same here. Monthly warning. What happens in case of paid account? Is there charge occurring after 100%?

…

On Mon 2 Oct 2023 at 10:48, mountainash ***@***.***> wrote: @robhawkes <https://github.com/robhawkes> did you ever hear back from Support? This is still an issue for me *(nearly 1 year later)*. — Reply to this email directly, view it on GitHub <#39133 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ4MWITSPU5OGQ3HNNBL53X5J5U5ANCNFSM6AAAAAASAUXVSY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mountainash]

That depends on your "Spending Limits" Settings for Codespaces. If you have it set at $0 - you won't be charged extra, but your access to Codespaces will be restricted. See Setting a spending limit

[Solido]

Ok so our account are not eligible for codespace. That’s a serious bug. Thanks 🙏

…

On Mon 2 Oct 2023 at 11:15, mountainash ***@***.***> wrote: That depends on your "Spending Limits" Settings for Codespaces. If you have it set at $0 - you won't be charged extra, but your access to Codespaces will be restricted. See Setting a spending limit <https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#setting-a-spending-limit> — Reply to this email directly, view it on GitHub <#39133 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ4MWPWJKH6VUJ6GQU6YS3X5KA4DANCNFSM6AAAAAASAUXVSY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mountainash]

It's not a bug... it's likely that you don't have payment details on your account, or your Organisation that your account is attached to, have disabled Billing for Codespaces... this is a feature.

What is a bug is that your replies to the email start a new comment each time instead of adding a reply to the existing comment in Github Discussions - please use the UI at https://github.com/orgs/community/discussions/39133 to keep the conversation clean.

[mountainash]

So this is not possible bug, it's the "intended" way that storage usage is calculated. See About billing for storage usage and read it a few times to attempt to understand it.

As the email alerts happen over different days, the calculated usage is compounding; in comparison to the monthly limit, has increased and therefore the remaining allocation is reduced more and more each day (hence the 75%, 90%, and 100% email alerts).

💡 I think the confusion is that the UI doesn't show you anywhere (that I can find) - how big your current container storage is for each Codespace. Even the downloadable usage report shows the gb-month which is the storage size (GB) divided by it's existence time (24 hrs * 30 days) - which adds to the confusion. Knowing how big a Space's storage (and prebuild) size is will allow users to make an informed decision on what to delete to stay within the included plan limits.

[TukangM]

and this buggy Codespaces calculation are still exist today

[nicolopadovan]

I am having a similar issue: every month, just a couple of days after the reset, my 15GB of Codespaces are chewed up to 90% (all by prebuilds).
I am NOT using Codespaces, and my activity on GitHub is primarily committing, pushing, pull requesting, and I have some projects set up for Vercel CI.