7 ms·

Show HN: Open-source x64 and Arm GitHub runners

475 points by umur 3y ago

Ubicloud is an open source cloud. Think of it as an open alternative to cloud providers, like what Linux is to proprietary operating systems. You can self-host Ubicloud or use our managed service. Our first use-case is GitHub Actions. We support x64 and arm64 Linux runners; and reduce your Github Actions bill by 10x. We can give you hardware similar to default GitHub runners because of the very high margins on the cloud. One difference with our hardware is that we use local NVMes for better disk performance. Ubicloud and our GitHub Actions integration is also open source. You can check out our integration here: https://github.com/ubicloud/ubicloud/blob/main/routes/web/webhook/github.rb https://github.com/ubicloud/ubicloud/blob/main/routes/web/we... For security and isolation, we give you a clean and ephemeral VM for each job. When the job completes, we deprovision the VM and wipe out the block storage device attached to the VM. We set up your firewall rules to lock down access to the VM; and also encrypt your data at rest and in-transit. (We use Linux KVM and the Cloud Hypervisor as our underlying VM tech. For our block device, we use and extend SPDK: https://www.ubicloud.com/blog/building-block-storage-for-cloud-with-spdk-non-replicated https://www.ubicloud.com/blog/building-block-storage-for-clo...) Ubicloud runners are also fully compatible with GitHub runners. To get started, all you need to do is change 1-line in your workflow file. Each account gets 1,250 free build minutes per month with standard-2 runners. We’d love to hear your feedback! https://www.ubicloud.com/docs/github-actions-integration/quickstart https://www.ubicloud.com/docs/github-actions-integration/qui...

deleted 3y ago
[deleted]
ozgune 3y ago
Hi there, I'm Ozgun, one of Ubicloud's founders. We have dozens of customers using Ubicloud runners in production today. We’re now designing our caching layer (Docker instance registry, Docker layer cache, or package cache). We wanted to put this out there for any comments. Also, if you have any points related to the broader topic of an open and portable cloud, please pass them along!
- maxmcd 3y ago
  Yes, please do what Depot does and put fast persistent disks close to builds to cache docker layers. Github action runners and circleci and all the others adding expensive network calls to manually cache layers has always been such a time sink and I think moves lots of people to remove caching entirely.
  - kylegalbraith 3y ago
    Depot [0] founder here. Thanks for the mention. We're also planning on bringing a bit of a different take to GitHub Action runners that's not tied to Hetzner directly. It will be entirely open-source as well, so you can take it and run it on your own instances if you'd like. Similar to how Depot supports self-hosted builders in your own AWS account [1]. [0] https://depot.dev/ https://depot.dev/ [1] https://depot.dev/docs/self-hosted/architecture https://depot.dev/docs/self-hosted/architecture
- maxloh 3y ago
  Could you upload your build of GitHub's runner image to Docker Hub? This would be quite useful for users of other GitHub Actions clones like act [0]. [0]: https://github.com/nektos/act https://github.com/nektos/act
kburman 3y ago
> wipe out the block storage device attached to the VM Does this provide guarantee that subsequent job won't be able to recover the data?
- auguzanellato 3y ago
  Well, it’s the same as regular GHA runners. On GitHub provided runners you need to explicitly move data between jobs. I find that useful in preventing weird side effects in one job affecting another.
- ozgune 3y ago
  Yes, a subsequent job won't be able to recover the data. We completely shut down the VM and remove the block device (all files associated with the block device).
  - riddley 3y ago
    I think the concern is that a subsequent allocation would have blocks from a previous allocation that could be readable.
- amenghra 3y ago
  If the data is also encrypted at rest (as they claim it is), then even if the raw data is recovered it shouldn't be useable without a combination of key leak.
  - fdr 3y ago
    I work at Ubicloud. Although we have a KEK and DEK code for regular VMs, they are not operative on GHA...yet. The reason has to do with a technical conflict with copy-on-write we aim to close, not least of which because Ubicloud needs to grow its own copy-on-write features for block device snapshots, things we lack today. I expect within a few months, all expired GHA vms will be cryptoshredded upon their deletion. This is already true for regular virtual machines or managed postgres machines.
hangonhn 3y ago
Our biggest GHA fees come from running on MacOS. Do you offer MacOS as a managed service (or plan to?) and how much cheaper is that than GitHub?
- HideousKojima 3y ago
  Biggest issue with MacOS is that it requires a physical Mac (not virtualization) and IIRC the licening from Apple requires a minimum "rental" period of 24 hours or something like that Edit: the TOS for OS X says this: 3. Leasing for Permitted Developer Services. A. Leasing. You may lease or sublease a validly licensed version of the Apple Software in its entirety to an individual or organization (each, a “Lessee”) provided that all of the following conditions are met: (i) the leased Apple Software must be used for the sole purpose of providing Permitted Developer Services and each Lessee must review and agree to be bound by the terms of this License; (ii) each lease period must be for a minimum period of twenty-four (24) consecutive hours;
  - yla92 3y ago
    What I did for work was, getting a Mac Mini and set up multiple self hosted runners in Tmux (very hackish, I know). The builds became faster (and cheaper) because each run no longer had to download the dependencies again. Of course, hosting myself means I also gotta own the uptime of it..
  - mysteria 3y ago
    How does Github Actions get away with this? They certainly don't bill and provide the machine for 24 hours for each CI run.
    - deleted 3y ago
      [deleted]
- ozgune 3y ago
  Not for the foreseeable future. Ubicloud runs on bare metal providers and they don't lease Mac hardware. Technically, we could run MacOS VMs on arm64. However, our interpretation of Apple's End User License Agreement (EULA) tells us that we can't do this. This repo has some good references on the topic: https://github.com/kholia/OSX-KVM?tab=readme-ov-file#is-this-legal https://github.com/kholia/OSX-KVM?tab=readme-ov-file#is-this...
forks 3y ago
How does this compare to BuildJet?
- gajus 3y ago
  given that the landing page looks almost pixel by pixel inspired by BuildJet, I'd say the answer is "very comparable"
- nateb2022 3y ago
  It's cheaper, for one
rgbrgb 3y ago
Congrats on the launch. Looks interesting. Quick thoughts on the landing page: - Pricing looks awesome. - I'm not currently the target audience because everything I'm doing right now is open source with free GitHub actions. - I'm left wondering what the catch is / why it's cheaper and faster. - Visual nit: lacking horizontal padding from 990px to ~1200px, a common window size on my 14" MBP. > Ubicloud is an open source cloud. Think of it as an open alternative to cloud providers, like what Linux is to proprietary operating systems. You can self-host Ubicloud or use our managed service. I find this hard to parse and the first few times I thought you were saying it's a linux alternative. I just clicked to the docs and the "What is Ubicloud?" section is clearer because you say concretely and directly what it is rather than how I should think of it metaphorically: "infrastructure-as-a-service (IaaS) features on providers that lease bare metal instances, such as Hetzner, OVH, and AWS Bare Metal. It’s also available as a managed service." There's some old "counterintuitive" adage I'm too lazy to look up about how the best marketing to engineers is just saying in concrete language what it is rather than the benefit it provides. In this case, I'd do both: tell me what it actually is and why that makes it cheaper/better. Also a minor note, there's a typo in that paragraph (missing space "systems.Ubicloud").
- Arelius 3y ago
  > I'm left wondering what the catch is / why it's cheaper and faster. So, while this is the first time I've heard of ubicloud, I use gha extensively. Abd frankly, I think it's just because Github has a crazy markup on their actions compute over the raw compute. Taking a quick look, it appears that their base rate is like $0.008 per-minute!! That's a rate that wouldn't look crazy out of line with EC2's hourly rate. I've worked on projects where we saved significant money, and improved build times just by launching a single EC2 instance and connecting it to Actions.
  - pinkgolem 3y ago
    Iops on GitHub runners is also terrible slow, you easily get a 5x to 10x improvement We did the same, and set up GitHub actions runners on hetzner Halfed the integration test time and made them more reliable.
justinzollars 3y ago
I always imagine the github runners as a vast array of early model raspberry pis.
JoshTriplett 3y ago
Sigh. Please don't call the Elastic license "open source". It's nice that the source is available, but this is not an open source license. EDIT: per responses, it looks like this is outdated information and the project now uses AGPL!
- matt_heimer 3y ago
  The docs still say the Elastic license is used but looking at https://github.com/ubicloud/ubicloud/blob/main/LICENSE https://github.com/ubicloud/ubicloud/blob/main/LICENSE it looks like the project might have switched to GNU Affero General Public License v3.0 in the last day.
  - umur 3y ago
    thank you and that's correct, just updated the docs as well.
    - JoshTriplett 3y ago
      That's wonderful news, thank you very much for switching to an actually open license! The linked page at https://www.ubicloud.com/docs/github-actions-integration/quickstart https://www.ubicloud.com/docs/github-actions-integration/qui... still says "Source open under the Elastic V2 license", and https://www.ubicloud.com/docs/about/pricing https://www.ubicloud.com/docs/about/pricing still says "it's open and free under the Elastic V2 license". Not sure if those were missed or if the docs just need some time to refresh from their sources.
    - mindwok 3y ago
      Wow. When you guys first launched this was my biggest concern. This is absolutely awesome, thank you team!
gajus 3y ago
Feels like there is another one of these every week.
- flir 3y ago
  Might be a sweet spot. I'm thinking (a) well-understood problem (b) hardware's rented, and scales with use (c) lots of teams have CI costs high enough to be annoying, but not so high that they need authorization to change their supplier (d) github are marking up the service so hard that it's easy to compete on price.
- risyachka 3y ago
  It is relatively easy to roll out a solution to this problem and GitHub doesn't care about competition in this niche.
- crohr 3y ago
  Yes, I wonder when/if GitHub decides to drop prices at some point. But in the meantime I’ve recently released RunsOn [1] with the same promise (10x cheaper, faster) but the whole thing runs in your AWS account. [1] https://runs-on.com https://runs-on.com
necubi 3y ago
We've been using Ubicloud builders for our Rust project [0] for several months, and it's worked very well. We've seen CI times go from 10-15 minutes to 6-7, and our bill has gone from $300/month to $30. One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build. [0] https://github.com/ArroyoSystems/arroyo https://github.com/ArroyoSystems/arroyo
- babanin 3y ago
  Thanks for sharing! I've been looking at the repository and noticed that some jobs are still running on Github hosted runners. What is the point of using them and not running everything on Ubicloud?
  - necubi 3y ago
    We have one very slow job (our Rust CI, for example: https://github.com/ArroyoSystems/arroyo/actions/runs/7702793822/job/20991814192 https://github.com/ArroyoSystems/arroyo/actions/runs/7702793...) and a bunch of little jobs that take a few seconds (checking lints, etc.). We never bothered to switch those over because they complete quickly on github and fit within our included runner minutes. I should say we also use BuiltJet for docker builds because of their arm support. Now that ubicloud has arm we may switch those over as well.
- csdvrx 3y ago
  > One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build. The link to SPDK was very interesting: https://www.ubicloud.com/blog/building-block-storage-for-cloud-with-spdk-non-replicated https://www.ubicloud.com/blog/building-block-storage-for-clo.... I use filesystems for very high performance applications, and I've found ZFS to often be the limiting factor when compared to simpler solutions of XFS +- mdadm +- encryption. It's a controversial point, but others have made similar findings: https://klarasystems.com/articles/virtualization-showdown-freebsd-bhyve-linux-kvm/ https://klarasystems.com/articles/virtualization-showdown-fr... : "Although I suspect this will surprise many readers, it didn’t surprise me personally—I’ve been testing guest storage performance for OpenZFS and Linux KVM for more than a decade, and zvols have performed poorly by comparison each time I’ve tested them" OpenZFS seems to be starting to consider optimizations to better perform on modern drivers (SSD, NVMe) which have very different performance profiles to what ZFS was built for (spinning rust) In the SPDK summary they say "To make VM provisioning times go faster, we changed our host OS from ext4 to btrfs" (...) "Also, when we switched the host filesystem to btrfs, our disk performance degraded notably. Our disk throughput dropped to about one-third of what it was with ext4." Ubicloud: the problem seems to be generic to CoW filesystems, and it's interesting you came with a slight variation (CoA) but have you considered the even simpler alternative of any journaling filesystems (XFS, Ext4...) with overlays? Or just UFS2 + snapshots to restore from a given state (initialized, ready for each test) then restore to this state between tests? I think customers finding that disabling cache works better means the CoA has similar issues to CoW. Personally, I'd have just tried to using SR-IOV with a namespace per customer, and call it a day instead of bringing extra complexity, but there must be good reasons for it. I'd love to know what these reasons are.
thomasisaac 3y ago
We've been happily using BuildJet [0] for over a year now. Saved over $25k in CI costs compared to GH Actions - they're also using super powerful bare metal servers with Hetzner - we got about a 94% reduction in build time! Absolutely chuffed, good to see more companies on the market though. [0] https://buildjet.com https://buildjet.com
acidhue 3y ago
Gitlab + Home laptop runners = free Been using this setup for year, very happy with it. I don't see the point of Github to be honest?
- belthesar 3y ago
  For what it's worth, you can also self host your runners without Ubicloud on Github. That doesn't remove other reasons to not run Gitlab, but it's also not unique. I've self hosted runners for a variety of reasons on Github with great success.
- dboreham 3y ago
  In a similar vein, using Gitea + self-hosted runners (including macos) here, very happy although we did do some of the work to make the "CI stack" (contributed back to gitea and act projects) so not totally batteries included yet. One thing that helps quite a bit imho is to avoid virtualization -- our approach is to run all CI jobs in containers, not VMs. Yes this has isolation implications, and requires some futzing to get docker-in-docker and docker-in-docker-in-docker to work (shout out to the Earthly team for figuring out how to host kind/k8s inside a container), but the "runs on any computer" property of containers (vs virtualization) is powerful. Want CI on your laptop? No problem. On a Windows machine? No problem.
roboben 3y ago
SOC2 ?
- tkellogg 3y ago
  are you pushing PHI/PII through github actions?
  - CSDude 3y ago
    Does not matter - pipeline needs to be trusted because it has access to sensitive resources for deployment tasks, can fake test results etc.
    - slekker 3y ago
      Even though it is a bit of a PITA to maintain self hosted runners, it is the reason we do it.
      - Klasiaster 3y ago
        GARM can easily manage ephemeral runners for you: https://github.com/cloudbase/garm https://github.com/cloudbase/garm (Ephemeral runners are also more secure)
  - manquer 3y ago
    Actions have access to environment secrets . Those secrets can open door to PII.
CSDude 3y ago
We've been running Ubicloud for a while at Resmo. It is indeed 10x cheaper. We upped our instance sizes by 2x for a slightly more performance but it's still 5x cheaper. The main reason is their platform is hosted on Hetzner dedicated instances.
- sscarduzio 3y ago
  So why don’t you have a GitHub action send a webhook request to your own CI in Hetzner?
deleted 3y ago
[deleted]
lpgauth 3y ago
One thing I find frustrating about the Github Actions's runner pricing is that it's calculated on a minute basis. Couldn't you bill by the second instead? Maybe set a minimum to 1 min but after that charge by the second? I assume this is done to cover the time that the VM reboots between jobs?
- suryao 3y ago
  This is exactly what we do at WarpBuild, more in the interest of fairness and not passing on random costs to users. The VM reboot times add up quickly and that's likely the rationale. However, it gets hard when there are users running linting jobs that take ~2s but running it with 16vcpu instances so we kept the 1 min floor.
timvdalen 3y ago
Seems like lots of people are thinking about this. We are rolling our own, and I spent a good chunk of today writing a new scheduler. Definitely fun to play around with.
fierro 3y ago
didn't see any mention of caching in the docs. This tends to be pretty important for fast build/CI at many different key points of a given workflow (dependency cache, build cache, docker layer cache, image cache, etc). Wondering if I missed something.
mastabadtomm 3y ago
Congrats on the launch! I hope Ubicloud will be even more successful than the previous project, Citus!
felipemesquita 3y ago
I am hearing about Ubicloud for the first time and it sounds very good. I don’t have a need for a cheaper GitHub runner now, but I’ve been dreaming about a modern, less complex open stack alternative for some time. Also, having a cloud service combined with the open source product seem like a great fit!
risyachka 3y ago
Am I missing something or GitHub can easily block all 3rd party runners if they want to?
- shepherdjerred 3y ago
  Of course, but doesn't GitHub embrace these runners?
  - prosim 3y ago
    Depends on how you define "embrace". Services like Ubicloud violate the GitHub Terms of Service.
    - LOLwierd 3y ago
      Can you elaborate on how and why?
commonenemy 3y ago
Looks like it's a long running node? Or am I wrong? Does it have on demand workers like those Kubernetes providers, where you use shared master nodes (or with a very small fee), and scalable node pools on demand (charged when used).
lijok 3y ago
Amazing, testing this out tomorrow !
DavyJone 3y ago
Congrats on the launch, certainly good to see competition. I was reading https://www.ubicloud.com/blog/ubicloud-hosted-arm-runners-100x-better-price-performance https://www.ubicloud.com/blog/ubicloud-hosted-arm-runners-10... but seems like a bit misleading to compare an ARM workload over QEMU vs a native ARM. What about native x86 vs native x86 or at least native x86 vs native arm?
syassami 3y ago
Looks neat. I've been using https://github.com/philips-labs/terraform-aws-github-runner https://github.com/philips-labs/terraform-aws-github-runner to save on $
cauchyk 3y ago
We've been using the Ubicloud runner for a while at PeerDB[1]. Great value and specially the ARM runners have been helpful to get our CI costs down. The team is really responsive and added the arm runner support within a few weeks of us requesting it. [1] https://github.com/PeerDB-io/peerdb https://github.com/PeerDB-io/peerdb
bastardoperator 3y ago
The people buying github/actions are getting at least 10K minutes included as part of their EA agreement, and they can use their own runners for zero cost. I think most enterprises are more concerned with having their bits on a cloud provider, versus pricing, but that's my limited experience.
cynix 3y ago
Will FreeBSD be supported in the future?
jovezhong 3y ago
Nice intro, but I am a bit worried not seeing SOC2 or something similar.
gitter101 3y ago
what stops GitHub from shutting down these offerings from a legal or technical perspective, these alternatives clearly violate their terms of service: "Additionally, regardless of whether an Action is using self-hosted runners, Actions should not be used for: the provision of a stand-alone or integrated application or service offering the Actions product or service, or any elements of the Actions product or service, for commercial purposes"
- Aeolun 3y ago
  I read that as them not wanting you to use Actions as part of your commercial offering. E.g. don’t use Github as Infrastructure as a Service.
- OJFord 3y ago
  No that means you can't create a CI/CD competitor that's 'hosted' in Actions. (e.g. Install OJFordCI GitHub app, pay at ojfordci.com/sign-up, view your CI results at ojfordci.com, but actually also at github.com in the Actions tab on your repo.) They absolutely support custom runners, it's how all these work, they don't need to stop them via ToS, they can just not allow it as an option. `runs-on: ubicloud` only works because GitHub implements it right.
Aeolun 3y ago
How does this pricing work? With purely spot based AWS runners we can barely reach the 10x cost treshold compared to github runners. Edit: Oh, Hetzner. They’re really ubiquitous in cheap computing these days.
- crohr 3y ago
  You can reach 10x cheaper with just AWS, especially on the larger runner sizes. That is even including disk costs. I put a calculator for RunsOn at https://runs-on.com/calculator/ https://runs-on.com/calculator/
  - Aeolun 3y ago
    I mean, I did just say that, but it seems to me they can’t just resell AWS spot instances because there’d be no margin left.
ospider 3y ago
I was looking for macOS M1 runner, which is not provided by GitHub. I'm willing to pay for that, but it seems that there are only Linux types now.
- suryao 3y ago
  WarpBuild[0] provides Apple Silicon macOS runners powered by M2 Pros. Note: I'm the founder. [0] https://docs.warpbuild.com/runners#macos-m2-pro-on-arm64 https://docs.warpbuild.com/runners#macos-m2-pro-on-arm64
- low_tech_punk 3y ago
  Launched today: https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/ https://github.blog/changelog/2024-01-30-github-actions-intr...
- fkorotkov 3y ago
  Cirrus Runners being doing it for a long time now: https://cirrus-runners.app/ https://cirrus-runners.app/
tmpfs 3y ago
This looks great and the pricing is obviously a big improvement, I will try it out for our Linux jobs. Sure would be good to see competition for the MacOS and Windows runners too, these are the ones that tend to cost us the most.
- suryao 3y ago
  We at WarpBuild currently support mac instances (M2 Pros). Windows support will come soon.
low_tech_punk 3y ago
Coincidence maybe? GitHub just launched M1 macOS machines today. https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/ https://github.blog/changelog/2024-01-30-github-actions-intr...
fermigier 3y ago
Where is your company incorporated? (e.g do you comply with the GDPR?). This should be stated on the website.
siwatanejo 3y ago
Hello, can I pay with bitcoin?
andag 3y ago
I want the vendor that can take care of this for us, but that can guarantee a private egress range, or can run things within our VPC. Several builds connect to internal resources, so running them on external nodes suboptimal and expensive when it comes to network egress.
- orsol 3y ago
  Not a vendor but I am heading in that direction. Drop me a line and I can help with that. My contact info is in my profile.
ipshii 3y ago
Thanks for sharing, looks cool! The one I have been using on all of my projects is https://cirun.io https://cirun.io they support all cloud providers and have been around for almost as long as GitHub Actions and I love the fact that you can use it with AWS spot instances, which costs me about less than $5 a month for t4g.large (running on demand).

475 points by umur 3y ago

deleted 3y ago

[deleted]

ozgune 3y ago

Hi there, I'm Ozgun, one of Ubicloud's founders. We have dozens of customers using Ubicloud runners in production today. We’re now designing our caching layer (Docker instance registry, Docker layer cache, or package cache). We wanted to put this out there for any comments. Also, if you have any points related to the broader topic of an open and portable cloud, please pass them along!

maxmcd 3y ago
Yes, please do what Depot does and put fast persistent disks close to builds to cache docker layers. Github action runners and circleci and all the others adding expensive network calls to manually cache layers has always been such a time sink and I think moves lots of people to remove caching entirely.
- kylegalbraith 3y ago
  Depot [0] founder here. Thanks for the mention. We're also planning on bringing a bit of a different take to GitHub Action runners that's not tied to Hetzner directly. It will be entirely open-source as well, so you can take it and run it on your own instances if you'd like. Similar to how Depot supports self-hosted builders in your own AWS account [1]. [0] https://depot.dev/ https://depot.dev/ [1] https://depot.dev/docs/self-hosted/architecture https://depot.dev/docs/self-hosted/architecture
maxloh 3y ago
Could you upload your build of GitHub's runner image to Docker Hub? This would be quite useful for users of other GitHub Actions clones like act [0]. [0]: https://github.com/nektos/act https://github.com/nektos/act

kburman 3y ago

> wipe out the block storage device attached to the VM Does this provide guarantee that subsequent job won't be able to recover the data?

auguzanellato 3y ago
Well, it’s the same as regular GHA runners. On GitHub provided runners you need to explicitly move data between jobs. I find that useful in preventing weird side effects in one job affecting another.
ozgune 3y ago
Yes, a subsequent job won't be able to recover the data. We completely shut down the VM and remove the block device (all files associated with the block device).
- riddley 3y ago
  I think the concern is that a subsequent allocation would have blocks from a previous allocation that could be readable.
amenghra 3y ago
If the data is also encrypted at rest (as they claim it is), then even if the raw data is recovered it shouldn't be useable without a combination of key leak.
- fdr 3y ago
  I work at Ubicloud. Although we have a KEK and DEK code for regular VMs, they are not operative on GHA...yet. The reason has to do with a technical conflict with copy-on-write we aim to close, not least of which because Ubicloud needs to grow its own copy-on-write features for block device snapshots, things we lack today. I expect within a few months, all expired GHA vms will be cryptoshredded upon their deletion. This is already true for regular virtual machines or managed postgres machines.

hangonhn 3y ago

Our biggest GHA fees come from running on MacOS. Do you offer MacOS as a managed service (or plan to?) and how much cheaper is that than GitHub?

HideousKojima 3y ago
Biggest issue with MacOS is that it requires a physical Mac (not virtualization) and IIRC the licening from Apple requires a minimum "rental" period of 24 hours or something like that Edit: the TOS for OS X says this: 3. Leasing for Permitted Developer Services. A. Leasing. You may lease or sublease a validly licensed version of the Apple Software in its entirety to an individual or organization (each, a “Lessee”) provided that all of the following conditions are met: (i) the leased Apple Software must be used for the sole purpose of providing Permitted Developer Services and each Lessee must review and agree to be bound by the terms of this License; (ii) each lease period must be for a minimum period of twenty-four (24) consecutive hours;
- yla92 3y ago
  What I did for work was, getting a Mac Mini and set up multiple self hosted runners in Tmux (very hackish, I know). The builds became faster (and cheaper) because each run no longer had to download the dependencies again. Of course, hosting myself means I also gotta own the uptime of it..
- mysteria 3y ago
  How does Github Actions get away with this? They certainly don't bill and provide the machine for 24 hours for each CI run.
  - deleted 3y ago
    [deleted]
ozgune 3y ago
Not for the foreseeable future. Ubicloud runs on bare metal providers and they don't lease Mac hardware. Technically, we could run MacOS VMs on arm64. However, our interpretation of Apple's End User License Agreement (EULA) tells us that we can't do this. This repo has some good references on the topic: https://github.com/kholia/OSX-KVM?tab=readme-ov-file#is-this-legal https://github.com/kholia/OSX-KVM?tab=readme-ov-file#is-this...

forks 3y ago

How does this compare to BuildJet?

gajus 3y ago
given that the landing page looks almost pixel by pixel inspired by BuildJet, I'd say the answer is "very comparable"
nateb2022 3y ago
It's cheaper, for one

rgbrgb 3y ago

Congrats on the launch. Looks interesting. Quick thoughts on the landing page: - Pricing looks awesome. - I'm not currently the target audience because everything I'm doing right now is open source with free GitHub actions. - I'm left wondering what the catch is / why it's cheaper and faster. - Visual nit: lacking horizontal padding from 990px to ~1200px, a common window size on my 14" MBP. > Ubicloud is an open source cloud. Think of it as an open alternative to cloud providers, like what Linux is to proprietary operating systems. You can self-host Ubicloud or use our managed service. I find this hard to parse and the first few times I thought you were saying it's a linux alternative. I just clicked to the docs and the "What is Ubicloud?" section is clearer because you say concretely and directly what it is rather than how I should think of it metaphorically: "infrastructure-as-a-service (IaaS) features on providers that lease bare metal instances, such as Hetzner, OVH, and AWS Bare Metal. It’s also available as a managed service." There's some old "counterintuitive" adage I'm too lazy to look up about how the best marketing to engineers is just saying in concrete language what it is rather than the benefit it provides. In this case, I'd do both: tell me what it actually is and why that makes it cheaper/better. Also a minor note, there's a typo in that paragraph (missing space "systems.Ubicloud").

Arelius 3y ago
> I'm left wondering what the catch is / why it's cheaper and faster. So, while this is the first time I've heard of ubicloud, I use gha extensively. Abd frankly, I think it's just because Github has a crazy markup on their actions compute over the raw compute. Taking a quick look, it appears that their base rate is like $0.008 per-minute!! That's a rate that wouldn't look crazy out of line with EC2's hourly rate. I've worked on projects where we saved significant money, and improved build times just by launching a single EC2 instance and connecting it to Actions.
- pinkgolem 3y ago
  Iops on GitHub runners is also terrible slow, you easily get a 5x to 10x improvement We did the same, and set up GitHub actions runners on hetzner Halfed the integration test time and made them more reliable.

justinzollars 3y ago

I always imagine the github runners as a vast array of early model raspberry pis.

JoshTriplett 3y ago

Sigh. Please don't call the Elastic license "open source". It's nice that the source is available, but this is not an open source license. EDIT: per responses, it looks like this is outdated information and the project now uses AGPL!

matt_heimer 3y ago
The docs still say the Elastic license is used but looking at https://github.com/ubicloud/ubicloud/blob/main/LICENSE https://github.com/ubicloud/ubicloud/blob/main/LICENSE it looks like the project might have switched to GNU Affero General Public License v3.0 in the last day.
- umur 3y ago
  thank you and that's correct, just updated the docs as well.
  - JoshTriplett 3y ago
    That's wonderful news, thank you very much for switching to an actually open license! The linked page at https://www.ubicloud.com/docs/github-actions-integration/quickstart https://www.ubicloud.com/docs/github-actions-integration/qui... still says "Source open under the Elastic V2 license", and https://www.ubicloud.com/docs/about/pricing https://www.ubicloud.com/docs/about/pricing still says "it's open and free under the Elastic V2 license". Not sure if those were missed or if the docs just need some time to refresh from their sources.
  - mindwok 3y ago
    Wow. When you guys first launched this was my biggest concern. This is absolutely awesome, thank you team!

gajus 3y ago

Feels like there is another one of these every week.

flir 3y ago
Might be a sweet spot. I'm thinking (a) well-understood problem (b) hardware's rented, and scales with use (c) lots of teams have CI costs high enough to be annoying, but not so high that they need authorization to change their supplier (d) github are marking up the service so hard that it's easy to compete on price.
risyachka 3y ago
It is relatively easy to roll out a solution to this problem and GitHub doesn't care about competition in this niche.
crohr 3y ago
Yes, I wonder when/if GitHub decides to drop prices at some point. But in the meantime I’ve recently released RunsOn [1] with the same promise (10x cheaper, faster) but the whole thing runs in your AWS account. [1] https://runs-on.com https://runs-on.com

necubi 3y ago

We've been using Ubicloud builders for our Rust project [0] for several months, and it's worked very well. We've seen CI times go from 10-15 minutes to 6-7, and our bill has gone from $300/month to $30. One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build. [0] https://github.com/ArroyoSystems/arroyo https://github.com/ArroyoSystems/arroyo

babanin 3y ago
Thanks for sharing! I've been looking at the repository and noticed that some jobs are still running on Github hosted runners. What is the point of using them and not running everything on Ubicloud?
- necubi 3y ago
  We have one very slow job (our Rust CI, for example: https://github.com/ArroyoSystems/arroyo/actions/runs/7702793822/job/20991814192 https://github.com/ArroyoSystems/arroyo/actions/runs/7702793...) and a bunch of little jobs that take a few seconds (checking lints, etc.). We never bothered to switch those over because they complete quickly on github and fit within our included runner minutes. I should say we also use BuiltJet for docker builds because of their arm support. Now that ubicloud has arm we may switch those over as well.
csdvrx 3y ago
> One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build. The link to SPDK was very interesting: https://www.ubicloud.com/blog/building-block-storage-for-cloud-with-spdk-non-replicated https://www.ubicloud.com/blog/building-block-storage-for-clo.... I use filesystems for very high performance applications, and I've found ZFS to often be the limiting factor when compared to simpler solutions of XFS +- mdadm +- encryption. It's a controversial point, but others have made similar findings: https://klarasystems.com/articles/virtualization-showdown-freebsd-bhyve-linux-kvm/ https://klarasystems.com/articles/virtualization-showdown-fr... : "Although I suspect this will surprise many readers, it didn’t surprise me personally—I’ve been testing guest storage performance for OpenZFS and Linux KVM for more than a decade, and zvols have performed poorly by comparison each time I’ve tested them" OpenZFS seems to be starting to consider optimizations to better perform on modern drivers (SSD, NVMe) which have very different performance profiles to what ZFS was built for (spinning rust) In the SPDK summary they say "To make VM provisioning times go faster, we changed our host OS from ext4 to btrfs" (...) "Also, when we switched the host filesystem to btrfs, our disk performance degraded notably. Our disk throughput dropped to about one-third of what it was with ext4." Ubicloud: the problem seems to be generic to CoW filesystems, and it's interesting you came with a slight variation (CoA) but have you considered the even simpler alternative of any journaling filesystems (XFS, Ext4...) with overlays? Or just UFS2 + snapshots to restore from a given state (initialized, ready for each test) then restore to this state between tests? I think customers finding that disabling cache works better means the CoA has similar issues to CoW. Personally, I'd have just tried to using SR-IOV with a namespace per customer, and call it a day instead of bringing extra complexity, but there must be good reasons for it. I'd love to know what these reasons are.

thomasisaac 3y ago

We've been happily using BuildJet [0] for over a year now. Saved over $25k in CI costs compared to GH Actions - they're also using super powerful bare metal servers with Hetzner - we got about a 94% reduction in build time! Absolutely chuffed, good to see more companies on the market though. [0] https://buildjet.com https://buildjet.com

acidhue 3y ago

Gitlab + Home laptop runners = free Been using this setup for year, very happy with it. I don't see the point of Github to be honest?

belthesar 3y ago
For what it's worth, you can also self host your runners without Ubicloud on Github. That doesn't remove other reasons to not run Gitlab, but it's also not unique. I've self hosted runners for a variety of reasons on Github with great success.
dboreham 3y ago
In a similar vein, using Gitea + self-hosted runners (including macos) here, very happy although we did do some of the work to make the "CI stack" (contributed back to gitea and act projects) so not totally batteries included yet. One thing that helps quite a bit imho is to avoid virtualization -- our approach is to run all CI jobs in containers, not VMs. Yes this has isolation implications, and requires some futzing to get docker-in-docker and docker-in-docker-in-docker to work (shout out to the Earthly team for figuring out how to host kind/k8s inside a container), but the "runs on any computer" property of containers (vs virtualization) is powerful. Want CI on your laptop? No problem. On a Windows machine? No problem.

roboben 3y ago

SOC2 ?

tkellogg 3y ago
are you pushing PHI/PII through github actions?
- CSDude 3y ago
  Does not matter - pipeline needs to be trusted because it has access to sensitive resources for deployment tasks, can fake test results etc.
  - slekker 3y ago
    Even though it is a bit of a PITA to maintain self hosted runners, it is the reason we do it.
    - Klasiaster 3y ago
      GARM can easily manage ephemeral runners for you: https://github.com/cloudbase/garm https://github.com/cloudbase/garm (Ephemeral runners are also more secure)
- manquer 3y ago
  Actions have access to environment secrets . Those secrets can open door to PII.

CSDude 3y ago

We've been running Ubicloud for a while at Resmo. It is indeed 10x cheaper. We upped our instance sizes by 2x for a slightly more performance but it's still 5x cheaper. The main reason is their platform is hosted on Hetzner dedicated instances.

sscarduzio 3y ago
So why don’t you have a GitHub action send a webhook request to your own CI in Hetzner?

deleted 3y ago

[deleted]

lpgauth 3y ago

One thing I find frustrating about the Github Actions's runner pricing is that it's calculated on a minute basis. Couldn't you bill by the second instead? Maybe set a minimum to 1 min but after that charge by the second? I assume this is done to cover the time that the VM reboots between jobs?

suryao 3y ago
This is exactly what we do at WarpBuild, more in the interest of fairness and not passing on random costs to users. The VM reboot times add up quickly and that's likely the rationale. However, it gets hard when there are users running linting jobs that take ~2s but running it with 16vcpu instances so we kept the 1 min floor.

timvdalen 3y ago

Seems like lots of people are thinking about this. We are rolling our own, and I spent a good chunk of today writing a new scheduler. Definitely fun to play around with.

fierro 3y ago

didn't see any mention of caching in the docs. This tends to be pretty important for fast build/CI at many different key points of a given workflow (dependency cache, build cache, docker layer cache, image cache, etc). Wondering if I missed something.

mastabadtomm 3y ago

Congrats on the launch! I hope Ubicloud will be even more successful than the previous project, Citus!

felipemesquita 3y ago

I am hearing about Ubicloud for the first time and it sounds very good. I don’t have a need for a cheaper GitHub runner now, but I’ve been dreaming about a modern, less complex open stack alternative for some time. Also, having a cloud service combined with the open source product seem like a great fit!

risyachka 3y ago

Am I missing something or GitHub can easily block all 3rd party runners if they want to?

shepherdjerred 3y ago
Of course, but doesn't GitHub embrace these runners?
- prosim 3y ago
  Depends on how you define "embrace". Services like Ubicloud violate the GitHub Terms of Service.
  - LOLwierd 3y ago
    Can you elaborate on how and why?

commonenemy 3y ago

Looks like it's a long running node? Or am I wrong? Does it have on demand workers like those Kubernetes providers, where you use shared master nodes (or with a very small fee), and scalable node pools on demand (charged when used).

lijok 3y ago

Amazing, testing this out tomorrow !

DavyJone 3y ago

Congrats on the launch, certainly good to see competition. I was reading https://www.ubicloud.com/blog/ubicloud-hosted-arm-runners-100x-better-price-performance https://www.ubicloud.com/blog/ubicloud-hosted-arm-runners-10... but seems like a bit misleading to compare an ARM workload over QEMU vs a native ARM. What about native x86 vs native x86 or at least native x86 vs native arm?

syassami 3y ago

Looks neat. I've been using https://github.com/philips-labs/terraform-aws-github-runner https://github.com/philips-labs/terraform-aws-github-runner to save on $

cauchyk 3y ago

We've been using the Ubicloud runner for a while at PeerDB[1]. Great value and specially the ARM runners have been helpful to get our CI costs down. The team is really responsive and added the arm runner support within a few weeks of us requesting it. [1] https://github.com/PeerDB-io/peerdb https://github.com/PeerDB-io/peerdb

bastardoperator 3y ago

The people buying github/actions are getting at least 10K minutes included as part of their EA agreement, and they can use their own runners for zero cost. I think most enterprises are more concerned with having their bits on a cloud provider, versus pricing, but that's my limited experience.

cynix 3y ago

Will FreeBSD be supported in the future?

jovezhong 3y ago

Nice intro, but I am a bit worried not seeing SOC2 or something similar.

gitter101 3y ago

what stops GitHub from shutting down these offerings from a legal or technical perspective, these alternatives clearly violate their terms of service: "Additionally, regardless of whether an Action is using self-hosted runners, Actions should not be used for: the provision of a stand-alone or integrated application or service offering the Actions product or service, or any elements of the Actions product or service, for commercial purposes"

Aeolun 3y ago
I read that as them not wanting you to use Actions as part of your commercial offering. E.g. don’t use Github as Infrastructure as a Service.
OJFord 3y ago
No that means you can't create a CI/CD competitor that's 'hosted' in Actions. (e.g. Install OJFordCI GitHub app, pay at ojfordci.com/sign-up, view your CI results at ojfordci.com, but actually also at github.com in the Actions tab on your repo.) They absolutely support custom runners, it's how all these work, they don't need to stop them via ToS, they can just not allow it as an option. `runs-on: ubicloud` only works because GitHub implements it right.

Aeolun 3y ago

How does this pricing work? With purely spot based AWS runners we can barely reach the 10x cost treshold compared to github runners. Edit: Oh, Hetzner. They’re really ubiquitous in cheap computing these days.

crohr 3y ago
You can reach 10x cheaper with just AWS, especially on the larger runner sizes. That is even including disk costs. I put a calculator for RunsOn at https://runs-on.com/calculator/ https://runs-on.com/calculator/
- Aeolun 3y ago
  I mean, I did just say that, but it seems to me they can’t just resell AWS spot instances because there’d be no margin left.

ospider 3y ago

I was looking for macOS M1 runner, which is not provided by GitHub. I'm willing to pay for that, but it seems that there are only Linux types now.

suryao 3y ago
WarpBuild[0] provides Apple Silicon macOS runners powered by M2 Pros. Note: I'm the founder. [0] https://docs.warpbuild.com/runners#macos-m2-pro-on-arm64 https://docs.warpbuild.com/runners#macos-m2-pro-on-arm64
low_tech_punk 3y ago
Launched today: https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/ https://github.blog/changelog/2024-01-30-github-actions-intr...
fkorotkov 3y ago
Cirrus Runners being doing it for a long time now: https://cirrus-runners.app/ https://cirrus-runners.app/

tmpfs 3y ago

This looks great and the pricing is obviously a big improvement, I will try it out for our Linux jobs. Sure would be good to see competition for the MacOS and Windows runners too, these are the ones that tend to cost us the most.

suryao 3y ago
We at WarpBuild currently support mac instances (M2 Pros). Windows support will come soon.

low_tech_punk 3y ago

Coincidence maybe? GitHub just launched M1 macOS machines today. https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/ https://github.blog/changelog/2024-01-30-github-actions-intr...

fermigier 3y ago

Where is your company incorporated? (e.g do you comply with the GDPR?). This should be stated on the website.

siwatanejo 3y ago

Hello, can I pay with bitcoin?

andag 3y ago

I want the vendor that can take care of this for us, but that can guarantee a private egress range, or can run things within our VPC. Several builds connect to internal resources, so running them on external nodes suboptimal and expensive when it comes to network egress.

orsol 3y ago
Not a vendor but I am heading in that direction. Drop me a line and I can help with that. My contact info is in my profile.

ipshii 3y ago

Thanks for sharing, looks cool! The one I have been using on all of my projects is https://cirun.io https://cirun.io they support all cloud providers and have been around for almost as long as GitHub Actions and I love the fact that you can use it with AWS spot instances, which costs me about less than $5 a month for t4g.large (running on demand).