10 ms·

I'll think twice before using GitHub Actions again

• baobun 2y ago
  GitHub Actions supporting yaml anchors would resolve one of the gripes, which I share. https://github.com/actions/runner/issues/1182 https://github.com/actions/runner/issues/1182
• bramblerose 2y ago
  In the end, this is the age old "I built by thing on top of a 3rd party platform, it doesn't quite match my use case (anymore) and now I'm stuck". Would GitLab have been better? Maybe. But chances are that there is another edge case that is not handled well there. You're in a PaaS world, don't expect the platform to adjust to your workflow; adjust your workflow to the platform. You could of course choose to "step down" (PaaS to IaaS) by just having a "ci" script in your repo that is called by GA/other CI tooling. That gives you immense flexibility but also you lose specific features (e.g. pipeline display).
  • perlgeek 2y ago
    > Would GitLab have been better? My impression of gitlab CI is that it's also not built for monorepos. (I'm a casual gitlab CI user).
    • dezgeg 2y ago
      I'm not sure if there's a monorepo vs polyrepo difference; just that anything complex is pretty painful in gitlab. YAML "programming" just doesn't scale.
  • Hamuko 2y ago
    Doesn't everything in GitLab go into a single pipeline? GitHub at least makes splitting massive CI/CD setups easier by allowing you to write them as separate workflows that are separate files.
    • dezgeg 2y ago
      You can have pipelines trigger child pipelines in gitlab, but usability of them is pretty bad, viewing logs/results of those always needs extra clicking.
    • dijksterhuis 2y ago
      > GitHub at least makes splitting massive CI/CD setups easier by allowing you to write them as separate workflows that are separate files. this makes me feel like you’re really asking “can i split up my gitlab CICD yaml file or does everything need to be in one file”. if that’s the case: yes it does eventually all end up in a single pipeline (ignoring child pipelines). but you can split everything up and then use the `include` statement to pull it all together in one main pipeline file which makes dealing with massive amounts of yaml much easier. https://docs.gitlab.com/ee/ci/yaml/includes.html https://docs.gitlab.com/ee/ci/yaml/includes.html you can also use `include` to pull in a yaml config from another project to add things like SAST on the fly. previous workplace i had like 4 CICD template repos and constructed all 30 odd actual build repos from those four templates. used `include` to pull in some yaml template jobs, which i made run when by doing something like (it’s been a while, might get this wrong) include: project: 'cicd/templates' file: 'builds.yml' stages: - build job_a: stage: build extends: .job_a_from_template variables: IMAGE_NAME: "myimage" IMAGE_REPO: "somerepo.org" this doesn’t run anything for `job_b_from_template` … you just end up defining the things you want to run for each case, plus any variables you need to provide / override. you can also override stuff like rules on when it should run if you want to. which is handy. gitlab CICD can be really modular when you get into it. if that wasn’t the case: on me. edit: switched to some yaml instead of text which may or may not be wrong. dunno. i have yet to drink coffee.
      • dijksterhuis 2y ago
        addendum you can also do something like this, which means you don’t have to redefine every job in your main ci file, just define the ones you don’t want to run include: project: 'cicd/templates' file: 'builds.yml' variables: IMAGE_NAME: something IMAGE_REPO: some.org job_b: rules: - when: never where the template you import has a job_a and job_b definition. both get pulled in, but job_b gets overwritten so it never runs. less useful when just splitting things into multiple files to make life simpler. super useful when using the same templates across multiple independent repositories to make everything build in as close to the same way as possible.
  • thayne 2y ago
    The problem is that your "ci" script often needs some information from the host system, like what is the target git commit? Is this triggered by a pull request, or a push to a branch? Is it triggered by a release? And if so, what is the version of the release? IME, much of the complexity in using Github Actions (or Gitlab CI, or Travis) is around communicating that information to scripts or build tools. That and running different tasks in parallel, and making sure everything you want passes.
• tevon 2y ago
  I call writing GitHub Actions "Search and Deploy", constantly pushing to a branch to get an action to run is a terrible pattern... You'd think, especially with the deep VS Code integration, they'd have at least a basic sanity-check locally, even if not running the full pipeline.
  • 8n4vidtmkvmk 2y ago
    Not just me then? I was trying to fix a GitHub action just today but I have no clue how I'm supposed to tear it, so I just keep making tiny changes and pushing.... Not a good system but I'm still within the free tier so I'm willing to put up with it I guess.
    • masklinn 2y ago
      I think it’s everyone, debugging GH actions is absolute hell, and it gets terrifying when the action interacts with the world (e.g. creating and deploying packages to a registry).
      • oefrha 2y ago
        > it gets terrifying when the action interacts with the world (e.g. creating and deploying packages to a registry). To be fair, testing actions with side effects on the wider world is terrifying even if you’re running it locally, maybe more so because your nonstandard local environment may have surprises (e.g. an env var you set then forgot) while the remote environment mostly only has stuff you set/installed explicitly, and you can be sloppier (e.g. accidentally running ./deploy when you wanted to run ./test). That part isn’t a GH Actions problem.
        • masklinn 2y ago
          Locally it is much easier to set up and validate test environments, or neuter some of the pipeline to test things out and ensure the rest produces expected results (in fact I usually dry-run by default and require a file or envvar to “real run”). Especially as some jobs (rightfully) refuse to run in PRs.
• androa 2y ago
  GitHub (Actions) is simply not built to support monorepos. Square peg in a round hole and all that. We've opted for using `meta` to simulate monorepos, while being able to use GitHub Actions without too much downsides.
  • Imustaskforhelp 2y ago
    hey could you please share the`meta` tool you mentioned , sounds interesting ! couldn't find it on internet [skill issue]
    • joshka 2y ago
      Guessing it's https://github.com/mateodelnorte/meta https://github.com/mateodelnorte/meta googlefu "meta github repo"
      • Imustaskforhelp 2y ago
        hey thanks! definitely interesting! I do wonder if this really solves the author problem because by the looks of it , you just have to run meta command and it would run over each of the sub directory. While at the same time , I think I like it because this is what I think people refer to as "modular monolith" Combining this with nats https://nats.io/ https://nats.io/ (hey if you don't want it to be over the network , you could use nats with the memory model of your application itself to reduce any overhead) and essentially just get yourself a really modular monolith in which you can then seperate things selectively (ahem , microservices) afterwards rather easily.
        • HumanOstrich 2y ago
          Modular monolith refers to the architecture of your application[1]. It's a different concept from "monorepo", although they can be used together. I'm not sure what NATS has to do with anything in this post or discussion. Also, a modular monolith is almost the antithesis of microservices. [1]: https://www.thoughtworks.com/en-us/insights/blog/microservices/modular-monolith-better-way-build-software https://www.thoughtworks.com/en-us/insights/blog/microservic...
• arghwhat 2y ago
  > no way of running actions locally My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, relying solely on one-liner build or test commands. If the task is more complicated than a one-liner, make a script for it in the repo to make it a one-liner. Doesn't matter if it's GitHub Actions, Jenkins, Azure DevOps (which has super cursed yaml), etc. This in turn means that you can do what the pipeline does with a one-liner too, whether manually, from a vscode launch command, a git hook, etc. This same approach can fix the mess of path-specific validation too - write a regular script (shell, python, JS, whatever you fancy) that checks what has changed and calls the appropriate validation script. The GitHub action is only used to run the script on PR and to prepare the CI container for whatever the script needs, and the same pipeline will always run.
  • azriel91 2y ago
    there's: https://github.com/Pernosco/gha-runner https://github.com/Pernosco/gha-runner but I'm not sure how complete it is, and probably doesn't satisfy the author's use cases
  • Tainnor 2y ago
    The reason why many CI configs devolve into such a mess isn't typically that they don't extract complicated logic into scripts, it's about all the interactions with the CI system itself. This includes caching, sharing of artifacts, generating reports, configuring permissions, ordering of jobs, deciding when which jobs will run, deciding what to do when jobs fail, etc. All of this can get quite messy in a large enough project.
    • pydry 2y ago
      It never becomes unbearably messy this way though. The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.
      • SOLAR_FIELDS 2y ago
• vrnvu 2y ago
  > GitHub doesn't care GitHub cares. GitHub cares about active users on their platform. Whether it's managing PRs, doing code reviews, or checking the logs of another failed action.
  • joshka 2y ago
    GitHub often actively doesn't act in situations where acting would be prudent, which portrays from an outside perspective a disinterest in those who give their time to document shortcomings. Would you care to guess when the last time that the GitHub API was updated? It's probably much longer than you'd think (2+ years at this point).
  • presentation 2y ago
    They don’t care about things that I care about, including everything the author talked about, and also things like allowing whitespace-ignore on diffs to be set on by default in a repo or per user - an issue that’s been open for half a decade now. (Whitespace is just noise in a typescript repo with automatic formatting) https://github.com/orgs/community/discussions/5486 https://github.com/orgs/community/discussions/5486
• makingstuffs 2y ago
  Not sure if I am missing something but you can definitely run (some?) GH actions locally with act: https://github.com/nektos/act https://github.com/nektos/act Seen a couple posts on here say otherwise.
  • joshdavham 2y ago
    He mentioned act in the article.
  • alhadrad 2y ago
    Act has limitations because GitHub Actions run via virtualization, while Act runs via containerization. This means that actions behave differently across the two platforms.
• flohofwoe 2y ago
  IMHO the main problem with GH Actions is that the runners are so slow. Feels like running your build on a frigging C64 sometimes ;)
  • Hamuko 2y ago
    Are you hosting your own runners or relying on GitHub's?
  • ramon156 2y ago
    Blacksmith is your buddy. Its free and just has better images for single-core operations. Unless you're Google, I can guarantee it's faster.
  • moorow 2y ago
    [dead]
  • rustd 2y ago
    GH hosted runners use shared hardware so the performance is never good. There are quite a few options available. Harness CI offers hyper optimized build infrastructure, paired with software intelligence (caching, running subset of tests based on the code change) can reduce build times up-to 4X compared to GH Actions.
• joshdavham 2y ago
  > It's a known thing that there is no way of running GitHub Actions locally. There is a tool called act but in my experience it's subpar. I really hope there will be a nice, official tool to run gh actions locally in the future. That would be incredible.
• benrutter 2y ago
  Oh boy, there's a special kind of hell I enter into everytime I set up new github actions. I wrote a blog post a few months ago about my pain[0] but one of the main things I've found over the years is you can massively reduce how horrible writing github actions is by avoiding prebuilt actions, and just using it as a handy shell runner. If you write behaviour in python/ruby/bash/hell-rust-if-you-really-want and leave your github action at `run: python some/script.py` then you'll have something that's much easy to test locally, and save yourself a lot of pain, even if you wind up with slightly more boilerplate. [0] https://benrutter.github.io/posts/github-actions/ https://benrutter.github.io/posts/github-actions/
  • Imustaskforhelp 2y ago
    theoretically we could also use https://just.systems/ https://just.systems/ or https://mise.jdx.dev/ https://mise.jdx.dev/ instead of directly calling gh actions but I haven't tried gh actions personally yet , If its really the nightmare you are saying , then that's sad.
  • riperoni 2y ago
    At this point, just pause with Github Actions and compare it to how GiLab handles CI. Much more intuitive, taking shell scripts and other script commands natively and not devolving into a mess of obfuscated typescript wrapped actions that need a shit ton of dependencies.
    • danillonunes 2y ago
      But you can do the same with GitHub, right? Although most docs and articles focus on 3rd party actions, nothing stops you to just run everything in your own shell script.
      • lolinder 2y ago
        Yes, you can, and we do at my current job. Much of the time it's not even really the harder approach compared to using someone else's action, it's just that the existence of third party actions makes people feel obliged to use them because they wouldn't want to be accused of Not Invented Here Syndrome.
• kjuulh 2y ago
  I am biased because I built the rust SDK for dagger. But I think it is a real step forward for CI. Is it perfect? Nope. But it allows fixing a lot of the shortcomings the author has. Pros: - pipeline as code, write it as golang, python, typescript or a mix of thr above. - Really fast once cached - Use your languages library for code sharing, versioning and testing - Runs everywhere local, ci etc. Easy to change from github actions to something else. Cons: - Slow on the first run. Lots of pulling of docker images - The DSL and modules can feel foreign initially. - Modules are definitely a framework, I prefer just building having a binary I can ship (which is why the rust SDK doesnt support modules yet). - Doesn't handle large mono repos well, it relies heavily on caching and currently runs on a single node. It can work if you don't have 100 of services especially if the builder is a large machine. Just the fact that you can actually write ci pipelines that can be tested, packaged, versioned etc. Allows us to ship our pipelines as products which is quite nice and something we've come to rely on heavily
  • electromech 2y ago
    I'm genuinely intrigued by Dagger, but also super confused. For example, this feels like extra complexity around a simple shell command, and I'm trying to grok why the complexity is worth it: https://docs.dagger.io/quickstart/test/#inspect-the-dagger-function https://docs.dagger.io/quickstart/test/#inspect-the-dagger-f... I'm a fanboy of Rust, Containerization, and everything-as-code, so on paper Dagger and your Rust SDK seems like it's made for me. But when I read the examples... I dunno, I just don't get it.
    • mdaniel 2y ago
      I'm merely an outsider to Dagger, but I believe the page you linked to would give one the impression "but why golang[1] around some shell literals?!" because to grok its value one must understand that m.BuildEnv(source) <https://docs.dagger.io/quickstart/env#inspect-the-dagger-function https://docs.dagger.io/quickstart/env#inspect-the-dagger-fun...> is programmatically doing what https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#example-of-jobsjob_idsteps https://docs.github.com/en/actions/writing-workflows/workflo... would do: define the docker image (if any), the env vars (if any), and other common step parameters That conceptually allows one to have two different "libraries" in your CI: one in literal golang, as a function which takes in a source and sets up common step configurations, and the other as Dagger Functions via Modules (<https://docs.dagger.io/features/modules https://docs.dagger.io/features/modules> or <https://docs.dagger.io/api/custom-functions#initialize-a-dagger-module https://docs.dagger.io/api/custom-functions#initialize-a-dag...>) which work much closer to GHA uses: blocks from a organization/repo@tag style setup with the grave difference that they can run locally or in CI, which for damn sure is not true of GHA uses: blocks The closest analogy I have is from GitLab CI (since AFAIK GHA does not allow yaml anchors nor "logical extension"): .common: &common # <-- use whichever style your team prefers image: node:21-slim cache: {} # ... environment: [] # ... my-job1: stage: test <<: *common script: [ npm, run, test:unit, run ] my-job2: stage: something-else extends: .common script: echo "do something else" 1: I'm aware that I am using golang multiple times in this comment, and for sure am a static typing fanboi but as the docs show they allow other languages, too
• bitliner2 2y ago
  Welcome to the jungle. https://medium.com/@bitliner/why-gitlab-can-be-a-pain-ae1aa62af10e https://medium.com/@bitliner/why-gitlab-can-be-a-pain-ae1aa6... I think it’s not only GitHub. Ideally we should handle it as any other code, that is: do tests, handle multiple environments including the local environment, lint/build time error detection etc
  • benrutter 2y ago
    Unhappy to confirm that for any poor souls using Azure DevOps, it's even worse.
  • rustd 2y ago
    Agree. CI should be handled as code.
• rednafi 2y ago
  You can't run AWS lambda or DyanmoDB locally too (well you can but it's a hassle). So by that logic, we shouldn't use them at all. I don't like working with CI too but I'll take GitHub Actions over Jenkins/CircleCI/TravisCI any day.
  • chriswarbo 2y ago
    > You can't run AWS lambda or DyanmoDB locally too (well you can but it's a hassle). So by that logic, we shouldn't use them at all. No, applying the logic to something like Lambda would mean implementing handlers like: function handle(lambdaRequest, lambdaContext) { return myFunction( stuffExtractedFromLambdaRequest(lambdaRequest), stuffExtractedFromLambdaContext(lambdaContext) ); } Then there's no need to go through the hassle of running Lambda functions locally; since we can just run `myFunction` locally instead. Dynamo isn't the same, since it's just a service/API that we call; we don't implement its logic, like we do for CI tasks, Lambda functions, etc. Whilst you're right that it's a hassle to run DynamoDB locally (although not too bad, in my experience); that's also not necessary. It's fine to run code locally which talks to a remote DynamoDB; just set up the credentials appropriately.
    • rednafi 2y ago
      Yeah, and that doesn't stop us from using either of them. What I tried to convey is that GHA isn't ideal and it has a few warts but it's still better than most of the options available out there.
  • sunshowers 2y ago
    The problem with the analogy is that GHA's interface is quite thick.
  • arccy 2y ago
    lambda: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/using-sam-cli-local.html https://docs.aws.amazon.com/serverless-application-model/lat... dynamodb: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html#docker https://docs.aws.amazon.com/amazondynamodb/latest/developerg... doesn't seem any harder than running any other db
• alkonaut 2y ago
  That GH Actions and Azure Pipelines both settled for this cursed Yaml is hard to understand. Just make a real programming language do it! And ffs make a local test env so I can run the thing.
• bhaney 2y ago
  Article title: "[Common thing] doesn't work very well!" Article body: "So we use a monorepo and-" Tale as old as time
  • rasso 2y ago
    I‘m also struggling with gh actions. And none of my repos is a monorepo.
  • deleted 2y ago
    [deleted]
• aa-jv 2y ago
  I use Github Actions as a fertile testing playground to work out how to do things locally. For example, if you've ever had to wade into the codesigning/notarization quagmire, observing the methods projects use with Github Actions to do it, can teach you a lot about how to do things, locally.
• pshirshov 2y ago
  > Jenkins, TeamCity Yeah-yeah, but it's not like they allow you to run your build definitions locally nor they address some other concerns. With GHA you may use nix-quick-install in a declarative manner, nixify your builds and then easily run them locally and under GHA. In case of jenkins/tc you would have to jump through much more hoops.
• rickette 2y ago
  Every CI system has its flaws but GitHub Actions in my opinion is pretty nice especially in terms of productivity; easy to setup, tons of prebuild actions, lots of examples, etc. I've used Tekton, Jenkins, Travis, Hudson, StarTeam, Rational Jazz, Continuum and a host of other CI systems over the years but GitHub Actions ain't bad.
• angoragoats 2y ago
  Why is this team sticking multiple directories that are “independent of each other” into a single repository? This sounds like a clear case of doing version control wrong. Monorepos come with their own set of challenges, and I don’t think there are many situations where they’re actually warranted. They certainly don’t help for completely independent projects.
  • cbare 2y ago
    Yeah, sounds like the problems are more due to monorepos rather than with GitHub actions. Seems like the pendulum always swings too far. Overdoing microservices results in redundant code and interservice spaghetti. Monorepos have their own set of issues. The only solution is to think carefully about a what size chunk of functionality you want to build, test, and deploy as a unit.
  • ripped_britches 2y ago
    No project within a single organization is completely independent. In general they all serve to meet a unified business objective and developers often need the global context on occasion. I used to be a multirepo proponent, but have fallen in love with Bazel and “everything bagel” repo.
    • angoragoats 2y ago
      > No project within a single organization is completely independent. In general they all serve to meet a unified business objective and developers often need the global context on occasion. Of course; I was only quoting the article. I am a firm believer in making things as simple as possible until you need something complicated, and monorepos/Bazel definitely get a “complicated” label from me.
• nunez 2y ago
  Posts like this make me miss Travis. Travis CI was incredible, especially for testing CI locally. (I agree with the author that act is a well done hack. I've stopped using it because of how often I'd have something pass in act and fail in GHA.) > GitHub doesn't care My take: GitHub only built Actions to compete against GitLab CI, as built-in CI was taking large chunks of market share from them in the enterprise.
  • sureIy 2y ago
    To be fair, GitHub also charges for Actions minutes and storage, so it's one of the few pieces that do generate revenue.
  • chubot 2y ago
    How so? I don’t recall this, and I used Travis, and then migrated to GitHub actions. As far as I can tell, they are identical as far as testing locally. If you want to test locally, then put as much logic in shell scripts as possible, decoupled from the CI.
  • homebrewer 2y ago
    Woodpecker supports running jobs on your own machine (and conveniently provides a command to do that for failed jobs), uses the same sane approach of passing your snippets to the shell directly (without using weird typescript wrappers), and is pluggable into all major forges, GitHub included.
• xinayder 2y ago
  I tried to use GitHub Actions on Forgejo and... It's so much worse than using an actual CI pipeline. With Woodpecker/Jenkins you know exactly what your pipeline is doing. With GitHub actions, not even the developers of the actions themselves know what the runner does.
  • lolinder 2y ago
    > use GitHub Actions on Forgejo What does this even mean? Are you talking about Forgejo Actions, or are you somehow hosting your code on a Forgejo instance but running CI through GitHub? > With Woodpecker/Jenkins you know exactly what your pipeline is doing. If you wrote it from the ground up, sure. On the other hand, I've inherited Jenkins pipelines that were written years before I got there and involved three to four different plugins, and they're way worse to work with than the GitHub Actions that I inherited.
    • xinayder 2y ago
      > What does this even mean? Are you talking about Forgejo Actions, or are you somehow hosting your code on a Forgejo instance but running CI through GitHub? yes, Forgejo Actions which is supposed to be a drop in replacement for GH Actions. You can say they're different things but the general idea and level of complexity is the same.
  • myaccountonhn 2y ago
    Best one I've used is the CI of sourcehut. So simple and so damn easy to set up.
    • gabeio 2y ago
      You basically achieve the same result on github actions if you just ignore all of the github action yaml “magic” settings in the syntax and let your makefile/script do the logic which also makes it trivial to debug locally. But upvote because I do love sourcehut, it’s just so clean!
      • 2y ago
• OptionOfT 2y ago
  So the way I've solved the multiple folders with independent checks is like this: all-done: name: All done # this is the job that should be marked as required on GitHub. It's the only one that'll reliably trigger # when any upstream fails: success # when all upstream skips: pass # when all upstream success: success # combination of upstream skip and success: success runs-on: ubuntu-latest needs: - calculate-version - cargo-build - cargo-fmt - cargo-clippy-and-report - cargo-test-and-report - docker-build - docker-publish if: | always() steps: - name: Fail! shell: bash if: | contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') run: | echo "One / more upstream failed or was cancelled. Failing job..." exit 1 - name: Success! shell: bash run: | echo "Great success!" That way it is resilient against checks not running because they're not needed, but it still fails when any upstream actually fails. Now, I did end up running the tests of the front-end and back-end because they upload coverage, and if my coverage tool doesn't get both, it'll consider it as a drop in coverage and fail its check. But in general, I agree with the writer of the post that it all feels like it's not getting enough love. For example, there is no support for yaml anchors, which really hampers reusability on things that cannot be extracted to separate flows (not to mention separate flows can only be nested 4 deep). There is also the issue that any commit made by GitHub actions doesn't trigger another build. This is understandable, as you want to avoid endless builds, but sometimes it's needed, and then you need to do the ugly workaround with a PAT (and I believe it can't even be a fine-grained one). Combine that with policies that set a maximum time limit on tokens, your build becomes brittle, as now you need to chase down the person with admin access. Then there is the issue of Docker actions. They tell you to pin the action to an sha to prevent replacements. Except the action itself points to a replaceable tag. Lastly, there is a bug where when you create a report for your action, you cannot specify the parent it belongs to. So your ESLint report could be made a child of your coverage report.
• keybored 2y ago
  Why is this so difficult? 1. We apparently don’t even have a name for it. We just call it “CI” because that’s the adjacent practice. “Oh no the CI failed” 2. It’s conceptually a program that reports failure if whatever it is running fails and... that’s it 3. The long-standing principle of running “the CI” after merging is so backwards that that-other Hoare disparagingly called the correct way (guard “main” with a bot) for The Not Rocket Science Principle or something. And that smug blog title is still used to this day (or “what bors does”) 4. It’s supposed to be configured declaratively but in the most gross way that “declarative” has ever seen 5. In the true spirit of centralization “value add”: the local option of (2) (report failure if failed) has to be hard or at the very least inconvenient to set up I’m not outraged when someone doesn’t “run CI”.
  • keybored 2y ago
    > We apparently don’t even have a name for it. We just call it “CI” because that’s the adjacent practice. “Oh no the CI failed” Martin Fowler apparently calls this “continuous build” (the build itself without CI necessarily). And that’s better.
• spzb 2y ago
  My recent experience with Github Actions is that it will randomly fail running a pipeline that hasn't changed with an incomprehensible error message. I re-run the action a few hours later and it works perfectly.
  • quesera 2y ago
    This is great. I also enjoy the "randomly and undebuggably hang until timeout" (25mins?) which is annoying and incomprehensible and costs money. Rerunning the same jobs always passes.
• forty 2y ago
  For the first point, some mono repo orchestrators (I'm thinking of at least pnpm) have a way to do : run all the (for example) tests for all the packages that had change from master branch + all packages that depend transitively from those packages. It's very convenient and avoid having to mess with the CI limitations on the matter
• zxor 2y ago
  > The problem is that this step will only run when I change something in the web-app1 folder. So if my pull request only made changes in api1 I will never be able to merge my pull request! This just seems like a bad implementation to me? There are definitely ways to set up your actions so that they run all of the unit tests without changes if you'd like, or so that api1's unit tests are not required for a web-app1 related PR to be merged.
  • hightrix 2y ago
    Absolutely correct. When creating a new workflow, I always disable push/pull_request triggered builds and instead use the manually triggered `workflow_dispatch` method. This makes testing a new workflow much easier. Additionally, you can use conditionals based on inputs in the `workflow_dispatch` meaning that you could easily setup a "skip api tests" or "include web tests" option.
    • verdverm 2y ago
      It sounds like they have the logic to skip certain things if nothing has changed. The problem is around pull request gates and the lack of dynamic "these tests must be passing before merging is allowed". There are setting on a repository in the ruleset / status checks area that are configured outside of the dynamic yaml of the GHA workflow
• SamuelAdams 2y ago
  > Our code sits in a monorepo which is further divided into folders. Every folder is independent of each other and can be tested, built, and deployed separately. If this is true, and you still have problems running specific Actions, why not break this into separate repositories?
  • Always42 2y ago
    My immediate response as well.
  • verdverm 2y ago
    There is a mono vs poly repo tradeoff. Pros & cons to each approach. If you are doing monorepo, it would be antithetical to break it up into the poly paradigm. You really don't want both
• ashishb 2y ago
  There are a lot of subtle pitfalls as well. Like no default timeouts, excess permissions etc. I wrote about it in detail https://ashishb.net/tech/common-pitfalls-of-github-actions/ https://ashishb.net/tech/common-pitfalls-of-github-actions/ And even created a tool to generate good configs http://github.com/ashishb/gabo http://github.com/ashishb/gabo
• spooneybarger 2y ago
  A lot of folks in this thread are focusing on the monorepo aspect of things. The "Pull request and required checks" problem exists regardless of monorepo or not. GitHub Actions allows you to only run checks if certain conditions are met, like "only lint markdown if the PR contains *.md files". The moment you decide to use such rules, you have the "Pull request and required checks" problem. No "monorepo" required. GitHub required checks at this time allow you to use with external services where GitHub has no idea what might run. For this reason, required checks HAVE to pass. There's no "if it runs" step. A required check on an external service might never run, or it might be delayed. Therefore, if GH doesn't have an affirmation that it passed, you can't merge. It would be wonderful if for jobs that run on GH where GH can know if the action is supposed to run, if required checks could be "require all these checks if they will be triggered". I have encountered this problem on every non-trivial project I use with GitHub actions; monorepo or not.
  • saxonww 2y ago
    This isn't really the problem, though. This is an easy problem to solve; the real problem is that it costs money to do so. Also: I'm not asserting that the below is good, just that it works. First, don't make every check a required check. You probably don't need to require that linting of your markdown files passes (maybe you do! it's an example). Second, consider not using the `on:<event>:paths`, but instead something like `dorny/paths-filter`. Your workflow now runs every time; a no-op takes substantially less than 1 minute unless you have a gargantuan repo. Third, make all of your workflows have a 'success' job that just runs and succeeds. Again, this will take less than 1 minute. At this point, a no-op is still likely taking less than 1 minute, so it will bill at 1 minute, which is going to be $.008 if you're paying. Fourth, you can use `needs` and `if` now to control when your 'success' job runs. Yes, managing the `if` can be tricky, but it does work. We are in the middle of a very large migration into GitHub Actions from a self-hosted GitLab. It was something we chose, but also due to some corporate choices our options were essentially GitHub Actions or a massive rethink of CI for several dozen projects. We have already moved into code generation for some aspects of GitHub Actions code, and that's the fifth and perhaps final frontier for addressing this situation. Figure out how to describe a graph and associated completion requirements for your workflow(s), and write something to translate that into the `if` statements for your 'success' jobs.
• p1necone 2y ago
  There's a workaround for the 'pull request and required check' issue. You create an alternative 'no op' version of each required check workflow that just does nothing and exits with code 0 with the inverse of the trigger for the "real" one. The required check configuration on github is just based off of job name, so either the trigger condition is true, and the real one has to succeed or the trigger condition is false and the no op one satisfies the PR completion rules instead. It seems crazy to me that such basic functionality needs such a hacky workaround, but there it is.
  • dayjaby 2y ago
    Or you can just check if the step was skipped. I don't get the point of the article. Managing a monorepo with acyclic dependencies is super easy: dornys path filter in one job and the other jobs check 1. whether their respective or any dependency's path got changed 2. and all dependency jobs were either successful or skipped. Done. No need to write an article.
• zzo38computer 2y ago
  I do not use GitHub Actions for these purposes, and if I did, I would want to ensure that it is a file that can run locally or whatever else just as well. I don't use GitHub Actions to prevent pull requests from being merged (I will always manage them manually), and do not use GitHub Actions to manage writing the program, for testing the program (it would be possible to do this, but I would insist on doing it in a way that is not vendor-locked to GitHub, and by putting most of the stuff outside of the GitHub Actions file itself), etc. I do have a GitHub Actions file for a purpose which is not related to the program itself; specifically, for auto-assignment of issues. In this case, it is clearly not intended to run locally (although in this case you could do so anyways if you could install the "gh" program on your computer and run the command mentioned there locally, but it is not necessary since GitHub will do it automatically on their computer). on: issues: types: - opened pull_request: types: - opened permissions: contents: read issues: write pull-requests: write jobs: default: runs-on: ubuntu-latest steps: - run: gh issue edit ${{ github.event.issue.number }} --add-assignee ${{ github.repository_owner }} env: GH_TOKEN: ${{ github.token }} GH_REPO: ${{ github.repository }}
• ryanisnan 2y ago
  One really interesting omission to this post is how the architecture of GitHub actions encourages (or at the very least makes deceivingly easy) making bad security decisions. Common examples are secrets. Organization or repository secrets are very convenient, but they are also massive security holes just waiting for unsuspecting victims to fall into. Repository environments have the ability to have distinct secrets, but you have to ensure that the right workflows can only access the right environments. It's a real pain to manage at scale. Being able to `inherit` secrets also is a massive footgun, just waiting to leak credentials to a shared action. Search for and leak `AWS_ACCESS_KEY_ID` anyone? Cross-repository workflow triggering is also a disaster, and in some circumstances you can abuse the differences in configuration to do things the source repository didn't intend. Other misc. things about GHA also are cool in theory, but fall down in practice. One example is the wait-timer concept of environments. If you have a multi-job workflow using the same environment, wait-timer applies to EACH JOB in the environment. So if you have a build-and-test workflow with 2 jobs, one for build, and one for test, each step will wait `wait-timer` before it executes. This makes things like multi-environment deployment pipelines impossible to use this feature, unless you refactor your workflows. Overall, I'd recommend against using GHA and looking elsewhere.
  • junto 2y ago
    Whilst I do detest much of Azure DevOps, one thing I do like about their pipelines is that we can securely use service connections and key vaults in Azure to secure pipeline tasks that require credentials to be managed securely.
  • maccard 2y ago
    What’s your suggestion for not-GHA?
  • wrboyce 2y ago
    While I do agree with you regarding encouraging bad secret management practices, one fairly nice solution I’ve landed on is using terraform to manage such things. I guess you could even take it a step further to have a custom lint step (running on GHA, naturally) that disallows secrets configured in a certain manner and blocks a deploy (again, on GHA) on failure. I guess what I’m saying is, it’s GHA all the way down.
• posix86 2y ago
  One thing that sounds very nice about Github are merge queues: Once your PR is ready, rather than merging, you submit it to the merge queue, which will rebase it on the last PR also on the merge queue. It then runs the CI on each PR, and finally merges them automatically once successful. If CI fails, doesn't get merged, and the next PR skips yours on the chain. Still a lot of computation & some wait time, but you can just click & forget. You can also parallelize it; since branches are rebased on each other, you can run CI in advance and, assuming your predecessor is also successful, reuse the result from yours. Only available for enterprise orgs though.
  • tialaramex 2y ago
    That sounds roughly like what happens for Rust. I write a Rust PR, somebody reviews it, they have feedback, I modify the PR, they're happy, and it passes to bors (originally: https://github.com/graydon/bors https://github.com/graydon/bors) which then tries to see whether this can be merged with Rust and if so does so. It is nice to know that if humans thought your change is OK, you're done. I've only committed small changes (compiler diagnostics, documentation) nothing huge, so perhaps if you really get up in it that's more work, but it was definitely a pleasant experience. ... and sure enough it turns out that work on one of the bors successors was in fact discontinued because you should just use this GitHub feature. TIL.
  • Kinrany 2y ago
    There is room for improvement: https://matklad.github.io/2023/06/18/GitHub-merge-queue.html https://matklad.github.io/2023/06/18/GitHub-merge-queue.html
  • kortilla 2y ago
    I remember when OpenStack had this a decade ago in open source software. How much the dream of OS has faded. :'( https://opensource.com/article/20/2/zuul https://opensource.com/article/20/2/zuul
• hinkley 2y ago
  Re: monorepo > In GitHub you can specify a "required check", the name of the step in your pipeline that always has to be green before a pull request is merged. As an example, I can say that web-app1 - Unit tests are required to pass. The problem is that this step will only run when I change something in the web-app1 folder. So if my pull request only made changes in api1 I will never be able to merge my pull request! Continuous Integration is not continuous integration if we don’t test that a change has no deleterious side effects on the rest of the system. That’s what integration is. So if you aren’t running all of the tests because they’re slow, then you’re engaging in false economy. Make your tests run faster. Modern hardware with reasonable test runners should be able to whack out 10k unit tests in under a minute. The time to run the tests goes up by a factor of ~7-10 depending on framework as you climb each step in the testing pyramid. And while it takes more tests to cover the same ground, with a little care you can still almost halve the run time replacing one test with a handful of tests that check the same requirement one layer down, or about 70% moving down two layers. One thing that’s been missing from most of the recent CI pipelines I’ve used is being able to see that a build is going to fail before the tests finish. The earlier the reporting of the failure the better the ergonomics for the person who triggered the build. That’s why the testing pyramid even exists.
  • rustd 2y ago
    Agreed, most of the CI tools don't help in getting feedback early to the developers. I shouldn't have to wait hours for my CI job to complete. Harness is a tool that can reduce build times by caching build artifacts, docker layers and only running a subset of tests that were impacted the by the code change.
  • maccard 2y ago
    I agree hardware should be that quick, but CI and cloud hardware is woefully underpowered unless you actively seek it out. I’ve also never seen a test framework spew out even close to that in practice. I’m not even sure most frameworks would do that with noop tests, which is sad.
    • hinkley 2y ago
• rustd 2y ago
  [flagged]
  • Celeo 2y ago
    Op is lead for this product.
• esafak 2y ago
  Where did the terrible idea of pipelines as config come from anyway?
• kazinator 2y ago
  > My team consists of about 15 engineers If it's not open source, I have no idea why you'd use GitHub at all. (And even then.) Keep your eggs in your own nest.
  • ripped_britches 2y ago
    What does this even mean?
• verdverm 2y ago
  Monorepos come with a different set of tradeoffs from polyrepos. Both have their pains. We have a similar setup with Jenkins, and have used CUE to tame a number of these issues. We did so by creating (1) a software catalog (2) per-branch config for versions and CI switches Similarly, we are adopting Dagger, more as part of a larger "containerize all of our CI steps" which works great for bringing parity to CI & local dev work. There are a number of secondary benefits and the TUI / WUI logs are awesome. Between the two, I have removed much of the yaml engineering in my work
• dalton_zk 2y ago
  One options its create your own CI, I think the others tools have pros/cons. This month I start to create to my team our own tool to build CI, I'm using go lang and create a webhook who call my API and apply what is need. I'm saying this because you can create the CI with your features.
• ripped_britches 2y ago
  My man/woman - you gotta try buildkite. It’s a bit more extra setup since you have to interface with another company, more API keys, etc. But when you outgrow GH actions, this is the way. Have used buildkite in my last two jobs (big US tech companies) and it has been the only pleasant part of CI.
  • habosa 2y ago
    +1 I've use Jenkins, Travis, Circle, Cirrus, GitHub Actions, and Buildkite. Buildkite is leagues ahead of all of the others. It's the only enjoyable CI system I've used.
  • cjk 2y ago
    This is indeed the way.
• ironfootnz 2y ago
  I’ve seen many teams get stuck when they rely too heavily on GitHub Actions’ magic. The key issue is how tightly your build logic and config become tied to one CI tool. If the declarative YAML gets too big and tries to handle complex branching or monorepos, it devolves into a maintenance headache—especially when you can’t test it locally and must push blind changes just to see what happens. A healthier workflow is to keep all the logic (build, test, deploy) in portable scripts and let the CI only orchestrate each script as a single step. It’s easier to troubleshoot, possible to run everything on a dev machine, and simpler if you ever migrate away from GitHub. For monorepos, required checks are maddening. This should be a first-class feature where CI can dynamically mark which checks apply on a PR, then require only those. Otherwise, you do hacky “no-op” jobs or you force your entire pipeline to run every time. In short, GitHub Actions can be powerful for smaller codebases or straightforward pipelines, but if your repo is big and you want advanced control, it starts to feel like you’re fighting the tool. If there’s no sign that GitHub wants to address these issues, it’s totally reasonable to look elsewhere or build your own thin orchestration on top of more flexible CI runners.
• yunusefendi52 2y ago
  I think devbox.sh would solve some of the issues, especially local development. You can also run devbox in CI
• habosa 2y ago
  Shameless plug but I built GitGuard (https://gitguard.dev https://gitguard.dev) to solve the "Pull request and required checks" problem mentioned here (and other problems). Basically: you set GitGuard as your required check and then write a simple GitGuard workflow like this: if anymatch(pull_files,"src/backend/.*") { assert(checkpassed("backend-tests")) } Email in my bio for anyone interested.
• webprofusion 2y ago
  GitHub Action has supported your own locally hosted runners for years, so I presume "there is no way of running GitHub Actions locally" is referring to something else.
• cjk 2y ago
  I have never used a CI system more flaky and slow than GitHub Actions. The one and only positive thing about it is that you get some Actions usage for free. The Azure machines GitHub uses for the runners by default have terrible performance in almost every regard (network, disk, CPU). Presumably it would be more reliable when using your own runners, but even the Actions control plane is flaky and doesn't always schedule jobs correctly. We switched to Buildkite at $DAYJOB and haven't looked back.
• Sparkyte 2y ago
  Blindly using automation or implementing it with validation will always bite a person in the butt. Been there done that. It is good but it should always be event driven with a point of user validation.
• mgaunard 2y ago
  assuming that every folder is independent sounds like bad design. if they're really independent out them in separate repos.
• robertritz 2y ago
  It's simple. Don't believe that the company purchased by Microsoft wants anything other than for you to use more compute.
  • edflsafoiewq 2y ago
    All OSS on Github gets free compute. I doubt they want you to waste it.
• aswerty 2y ago
  I once used Team City and Octopus Deploy in a company. And ever since then, dealing with Gitlab Pipelines and Github Actions, I find them so much poorer as a toolkit. We are very much in the part of the platform cycle where best-in-breed is losing out to all-in-one. Hopefully we see things swing in the other direction in the next few years where composable best-in-breed solutions recapture the hearts and minds of the community.
• fuzzy2 2y ago
  I hate GitHub Actions, and I hate Azure Pipelines, which are basically the same. I especially hate that GitHub Actions has the worst documentation. However, I’ve come full circle on this topic. My current position is that you must go all-in with a given CI platform, or forego the benefits it offers. So all my pipelines use all features, to offer a great experience for devs relying on them: Fast, reproducible, steps that are easy to reason about, useful parameters for runs, ...
• the_gipsy 2y ago
  In my newest hobby project, I decided to bite the bullet and use the flake.nix as single source of truth. And it's surprisingly fast! I used cargo-crane to cache rust deps. This also works locally just running "nix flake check". Much better than dealing with github actions, caches, and whatnot. Apart from the nix gh action that just runs "nix flake check", the only other actions are for making a github release on certain tags, and uploading release artifacts - which is something that should be built-in IMO.
• jcarrano 2y ago
  The general philosophy of these CI systems is flawed. Instead of CI running your code, your code should run the CI. In other words, the CI should present an API such that one can have arbitrary code which informs the system of what is going on. E.g. "I'm starting jobs A,B,C", "Job A done successfully", "This file is an artifact for job B". Information should only from from the user scripts to the CI, and communication should be done by creating files in a specific format and location. This way the system can run and produce the same results anywhere provided it has the right environment/container.
• theknarf 2y ago
  Just have Github Actions run a monorepo tool like turborepo. You're just trying to do to much in a yaml file... The solution for all build pipeline tools are always to do most of your build logic in a bash-script/makefile/monorepo-tool.
• jpgvm 2y ago
  Use Bazel. GHA/Gitlab CI/Buildkite/whatever else floats your boat then just builds a bunch of Bazel targets, naively, in-order etc. Just lean on Bazel fine-grained caching until that isn't enough anymore and stir in remote build execution for more parallelism when you need it. This works up until ~10M+ lines of code or ~10ish reasonably large services. After that you need to do a bit more work to only build graph of targets that have been changed by the diff. That will get you far enough that you will have a whole team that works on these problems. Allowing the CI tools to do any orchestration or dictate how your projects are built is insanity. Expressing dependencies etc in YAML is is the path to darkness and is only really justifiable for very small projects.
  • pnathan 2y ago
    I have moved to this. CI works _fantastic_. Absolute top-notch. Then I have to sort out deployment...
• manx 2y ago
  I recommend to try earthfiles: https://earthly.dev/earthfile https://earthly.dev/earthfile This basically brings docker layer caching to CI. Only Things that changed are rebuilt and tested.
• TigerC10 2y ago
  Google made Release-Please to make monorepo development easier, there is a GitHub Action for it in the marketplace. Would probably make things a lot cleaner for this situation.
• melezhik 2y ago
  Many if not all mentioned issues derive from the fact that nowadays pipelines are most of the time - YML based - which is terrible choise for programming , you might want take a look at Sparky which is 100% Raku cicd system thst does not have many of mentioned pitfalls and super flexible … Disclaimer I am the tool author - https://github.com/melezhik/sparky https://github.com/melezhik/sparky