7 ms·

Seeing how much people recommend other solutions, I've actually moved from Travis to Jenkins, and never looked back. Yes, Jenkins has its issues (crappy UX, po

by tzaman 10y ago

Seeing how much people recommend other solutions, I've actually moved from Travis to Jenkins, and never looked back. Yes, Jenkins has its issues (crappy UX, poor/awkward docs), but where it shines is the fact it's self-hosted, so I can SSH onto the instance to debug a failing build or replay it with a modified Jenkinsfile on the fly. I'm quite proud of the current setup we have; We're hosting our app with Google's Container Engine (Kubernetes), so what we're doing is on every build Jenkins creates a slave agent within the same cluster (just different node pool) as the production, so the environment in which test containers are ran is identical to production, and what's more it actually talks to the same Kubernetes master, which means I can, for example, test our Nginx Reverse proxy with real backend endpoints and real certificates (that are mounted through glusterfs).

knocte 10y ago
IMO moving from Travis to Jenkins seems like a very disruptive change. As for me, I had used Travis just a bit, and same for Jenkins, and didn't like any of those options very much. So when my team needed to setup a CI solution, we ended up using GitLab CI, and it brings the best of both worlds: - Free service version if you use the GitLab.com deployment (granted, gitlab.com is a bit slow because it's the new thing and everybody is using it now), like travis. - Open source so that you can host it yourself in the future if you need to, like Jenkins. - Easy to use and configure, like Travis. - Free service for private repos in GitLab.com (neither Travis nor any Jenkins-service provider offer this, AFAIK). I plan to never look back.
- sytse 10y ago
  Thanks for your kind words about GitLab. We're working hard on making GitLab.com faster, see https://gitlab.com/gitlab-org/gitaly/issues/ https://gitlab.com/gitlab-org/gitaly/issues/ for the latest. A workaround would be to run your own GitLab installations, all the CI features are in the open source GitLab CE https://about.gitlab.com/products/#comparison https://about.gitlab.com/products/#comparison For an intro to Gitlab CI see https://about.gitlab.com/gitlab-ci/ https://about.gitlab.com/gitlab-ci/
- tzaman 10y ago
  I'll definitely check it out - but the problem with many of these tools is a lack of "proper" Docker support, along with the ability to self host and debug. Thanks for the heads up!
  - sytse 10y ago
    GitLab CI has proper Docker support. You can set a default docker image for each runner `--docker-image ruby:2.1` or set one for the project in the .gitlab-ci.yml file. For more information see https://docs.gitlab.com/ce/ci/docker/using_docker_images.html https://docs.gitlab.com/ce/ci/docker/using_docker_images.htm... Did you know GitLab also comes with a private container registry? https://about.gitlab.com/2016/05/23/gitlab-container-registry/ https://about.gitlab.com/2016/05/23/gitlab-container-registr...
    - tzaman 10y ago
      Thanks for clarification - I'll definitely have a look :)
    - aeharding 10y ago
      Also notable is services like docker-in-docker (and privileged docker containers) are allowed, which is a huge win over services like Atlassian's Pipelines. Without dind, it's really hard/annoying to us CI/CD to build docker images. But with Gitlab, they also provide a place to store docker images (the "registry") right next to your code, for free! At work, we're investigating moving to Gitlab for everything except issues (which we'd need to keep on JIRA for now since we're so embedded with it). It looks like Gitlab does have some integration with JIRA, but it's project-level, and it would be nice if it could be group-level since we have many small repos. :) (I don't have an affiliation with Gitlab, I'm just super happy with the service.) EDIT: typo
      - sytse 10y ago
        Thanks for your kind words for GitLab! Our JIRA support is pretty extensive https://docs.gitlab.com/ee/project_services/jira.html https://docs.gitlab.com/ee/project_services/jira.html but extending it further is a priority. Felipe Artur is working on this full time. Consider creating an issue for group level integration, it sounds interesting.
- dom0 10y ago
  Honestly my experience is that SaaS / hosted CI is just generally annoying due to constant stability and performance (both transient and in general, eg. caching in Travis) issues. Self-hosted CI on the other hand requires a lot of resources, especially when you're not only testing Linux or BSD, but proprietary OSes (OSX, Windows, although the latter at least has the Edge images that work everywhere). Ultimately I feel like the use of CI in many open source projects isn't very high due to constant annoyances and difficulty debugging the CI environment.
  - aeharding 10y ago
    Just a note - you can host your own CI node (runner) for project(s) on gitlab.com. It's super simple to setup, too. https://docs.gitlab.com/runner/install/linux-repository.html https://docs.gitlab.com/runner/install/linux-repository.html
    - drdaeman 10y ago
      This doesn't solve issues in runner-to-base comms. I had tried this kind of setup relatively recently, and almost every 10-15th build had failed for me because GitLab wasn't healthy (502 from Docker registry or problems up/downloading artifacts between stages, etc.) Fully self-hosted looks like the only sane way. Harder to set up, but at least one can check the whole chain that way.
      - sytse 10y ago
        Sorry about GitLab.com having performance problems. Self hosted is indeed the only way to work around it in the short term. I hope you can set it up quickly. Please use the Omnibus packages that should install in a few minutes https://about.gitlab.com/installation/ https://about.gitlab.com/installation/
  - pselbert 10y ago
    > Ultimately I feel like the use of CI in many open source projects isn't very high due to constant annoyances and difficulty debugging the CI environment. My perspective may be driven by the communities I'm involved in, but this seems wrong to me. From my experience I would say most open source projects use CI, particularly because so many of the bigger platforms (Travis, Circle, CodeShip) make it free for open source. It is an extremely valuable resource when you consider the infrastructure and integration you get with absolutely no effort. Installing more obscure libraries or cutting edge releases can be a pain, but it is minuscule compared to the time it would take to provision, secure, and manage a group of containers for an open source project.
- sambe 10y ago
  Do you really find Travis easy to use and configure? I've used TeamCity a lot and find the functionality to be decent to good but the UI horrible. Perhaps a bit biased because I had access to the server and agents in that case. A lot of debugging/discoverability issues are easier with full access of course. I've then used Travis a bit for small GitHub projects (and contributing to other people's projects) and it has a nice UI for simple stuff. However, it gives the impression that they are struggling to stay alive - many features are in beta or feel like they are; a lot is under-documented or planned soon (for multiple years). I know I can dig into their various projects' source code to find out how things work, but I always feel incredibly unproductive figuring something out in Travis. I worry for them. Great to hear you have a good experience with GitLab!
sandGorgon 10y ago
how are you deploying secrets to your kuberenetes CI pipeline ? for example on a build, you create (I'm assuming) a new namespace. it seems you are using glusterfs for your persistentvolumeclaim.. why not gce-pd ?
- tzaman 10y ago
  Juenkins' Kubernetes plugin allows to mount secrets like with usual pods so we deploy test secrets just like we normally do production ones. As for glusterfs, our Nginx reverse proxy also has a cron job that runs letsencrypt automatic renewal once a week so it needs to be able to write those new certificates. Because we need to be able to run several reverse proxies, all with write persmissions (we run one at a time, but on deployment, there are two running to avoid downtime) we chose gluster, as gce-pd doesn't support multiple writers.
  - sandGorgon 10y ago
    Oh wow. I have never used glusterfs on GCE - how difficult was it to set up and manage it ? What about ceph ?
Kudos 10y ago
FWIW, Circle CI offers the ability to SSH in to build nodes and free nodes for public projects
- fredley 10y ago
  It's really well implemented too, it grabs authorised public keys from the github repo directly, so if you can push to the repo for the project, you can just ssh straight into the build machine. Magic.
- gurgus 10y ago
  Was looking for this comment! CircleCI is amazing. We use it and it seems to offer us everything we need in a very nice shiny box.
- haldean 10y ago
  ...sort of. I eventually rage-quit Circle and set up a Jenkins cluster because of all of the heisenbugs we found on Circle. Builds would fail 5-10% of the time for totally unreproducible reasons (for example, pip install into a venv would fail with a permission error), and you can't SSH into a build that's already failed. We very rarely had problems with Jenkins builds, and when we did, we could go look at the environment it had run in and diagnose what went wrong. I love Jenkins and would absolutely choose it over a hosted solution. (We also went from paying $1k/mo to $0/mo, which is a very nice side effect)
  - tzaman 10y ago
    > (We also went from paying $1k/mo to $0/mo, which is a very nice side effect) That's not entirely true, because someone had to set Jenkins up and has to maintain it, but once things are rolling, it hardly needs any input.
  - pm90 10y ago
    Agree completely. I think new users are perhaps turned off by Jenkins due to its (deserved) reputation for ugly UI. But its very dependable, stable and reliable solution. And there are plugins for doing every conceivable thing. We have a job manager to store the config in CI, and its such a breeze to work with.
scrollaway 10y ago
The problem with travis and the like is most do not support arbitrary builds. Drone for example is trying to sell itself as a "jenkins replacement" but has no concept of a build triggering arbitrarily, or that isn't intrinsically linked to a git repo. It's nonsense. Once you set up a bunch of tooling jobs on Jenkins it's very nice to be able to use it as some form of control center for a bunch of different operations. Calling it a build server is underselling it.
- sytse 10y ago
  I think there are two aspects to arbitrary builds: 1. Periodic jobs (not linked to a pushed commit) 2. Not linked to a git repo At GitLab we try to have 'infrastructure as code'. We think builds should be under version control to allow for collaboration. But periodic jobs make a lot of sense and it is discussed in https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 https://gitlab.com/gitlab-org/gitlab-ce/issues/2989
  - jacques_chester 10y ago
    I think one thing GitlabCI lost in taking inspiration from Concourse (amongst other tools!) was the centrality of resources. It wasn't obvious at first why this is so important, I and most others fixated on the other visible differences (containers for everything, declarative config, pipeline view etc). There are a lot of features that don't need to be added to the core because they can be resources instead. For git-triggered builds, I use the git resource. Periodic, I use the time resource or a cron resource. S3 triggered, I use the S3 resource. Triggered on release of new software to PivNet, I use pivnet-resource. Triggered on a new docker image, I use docker image resource. And so on. Any resource that has a 'get' operation can trigger a build. 3rd parties can add triggers without needing to avoid interference with other resources, because every operation is isolated. Disclosure: I work for Pivotal, which sponsors Concourse.
    - sytse 10y ago
      Concourse is great and it was certainly an inspiration for GitLab CI. What I liked most and what we're still working on are cross project triggers https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 The use of resources sounds cool.
arekkas 10y ago
Are you running test code in production environment? Not sure about this one
- tzaman 10y ago
  I wasn't at first either, but benefits outweigh the risks (which are all security related). It's something I'm willing to re-think if it turns out to be problematic on any level. But so far after half a year in production, we've seen no problems.
donatj 10y ago
We run Drone and are very happy with it. Really easy to install and get working, similar but better usage than Travis, open source, Docker powered. I really hope it catches on more strongly because it's fantastic.
chrisabrams 10y ago
We are doing the exact same thing and also couldn't be happier :)

by tzaman 10y ago

knocte 10y ago

IMO moving from Travis to Jenkins seems like a very disruptive change. As for me, I had used Travis just a bit, and same for Jenkins, and didn't like any of those options very much. So when my team needed to setup a CI solution, we ended up using GitLab CI, and it brings the best of both worlds: - Free service version if you use the GitLab.com deployment (granted, gitlab.com is a bit slow because it's the new thing and everybody is using it now), like travis. - Open source so that you can host it yourself in the future if you need to, like Jenkins. - Easy to use and configure, like Travis. - Free service for private repos in GitLab.com (neither Travis nor any Jenkins-service provider offer this, AFAIK). I plan to never look back.

sytse 10y ago
Thanks for your kind words about GitLab. We're working hard on making GitLab.com faster, see https://gitlab.com/gitlab-org/gitaly/issues/ https://gitlab.com/gitlab-org/gitaly/issues/ for the latest. A workaround would be to run your own GitLab installations, all the CI features are in the open source GitLab CE https://about.gitlab.com/products/#comparison https://about.gitlab.com/products/#comparison For an intro to Gitlab CI see https://about.gitlab.com/gitlab-ci/ https://about.gitlab.com/gitlab-ci/
tzaman 10y ago
I'll definitely check it out - but the problem with many of these tools is a lack of "proper" Docker support, along with the ability to self host and debug. Thanks for the heads up!
- sytse 10y ago
  GitLab CI has proper Docker support. You can set a default docker image for each runner `--docker-image ruby:2.1` or set one for the project in the .gitlab-ci.yml file. For more information see https://docs.gitlab.com/ce/ci/docker/using_docker_images.html https://docs.gitlab.com/ce/ci/docker/using_docker_images.htm... Did you know GitLab also comes with a private container registry? https://about.gitlab.com/2016/05/23/gitlab-container-registry/ https://about.gitlab.com/2016/05/23/gitlab-container-registr...
  - tzaman 10y ago
    Thanks for clarification - I'll definitely have a look :)
  - aeharding 10y ago
    Also notable is services like docker-in-docker (and privileged docker containers) are allowed, which is a huge win over services like Atlassian's Pipelines. Without dind, it's really hard/annoying to us CI/CD to build docker images. But with Gitlab, they also provide a place to store docker images (the "registry") right next to your code, for free! At work, we're investigating moving to Gitlab for everything except issues (which we'd need to keep on JIRA for now since we're so embedded with it). It looks like Gitlab does have some integration with JIRA, but it's project-level, and it would be nice if it could be group-level since we have many small repos. :) (I don't have an affiliation with Gitlab, I'm just super happy with the service.) EDIT: typo
    - sytse 10y ago
      Thanks for your kind words for GitLab! Our JIRA support is pretty extensive https://docs.gitlab.com/ee/project_services/jira.html https://docs.gitlab.com/ee/project_services/jira.html but extending it further is a priority. Felipe Artur is working on this full time. Consider creating an issue for group level integration, it sounds interesting.
dom0 10y ago
Honestly my experience is that SaaS / hosted CI is just generally annoying due to constant stability and performance (both transient and in general, eg. caching in Travis) issues. Self-hosted CI on the other hand requires a lot of resources, especially when you're not only testing Linux or BSD, but proprietary OSes (OSX, Windows, although the latter at least has the Edge images that work everywhere). Ultimately I feel like the use of CI in many open source projects isn't very high due to constant annoyances and difficulty debugging the CI environment.
- aeharding 10y ago
  Just a note - you can host your own CI node (runner) for project(s) on gitlab.com. It's super simple to setup, too. https://docs.gitlab.com/runner/install/linux-repository.html https://docs.gitlab.com/runner/install/linux-repository.html
  - drdaeman 10y ago
    This doesn't solve issues in runner-to-base comms. I had tried this kind of setup relatively recently, and almost every 10-15th build had failed for me because GitLab wasn't healthy (502 from Docker registry or problems up/downloading artifacts between stages, etc.) Fully self-hosted looks like the only sane way. Harder to set up, but at least one can check the whole chain that way.
    - sytse 10y ago
      Sorry about GitLab.com having performance problems. Self hosted is indeed the only way to work around it in the short term. I hope you can set it up quickly. Please use the Omnibus packages that should install in a few minutes https://about.gitlab.com/installation/ https://about.gitlab.com/installation/
- pselbert 10y ago
  > Ultimately I feel like the use of CI in many open source projects isn't very high due to constant annoyances and difficulty debugging the CI environment. My perspective may be driven by the communities I'm involved in, but this seems wrong to me. From my experience I would say most open source projects use CI, particularly because so many of the bigger platforms (Travis, Circle, CodeShip) make it free for open source. It is an extremely valuable resource when you consider the infrastructure and integration you get with absolutely no effort. Installing more obscure libraries or cutting edge releases can be a pain, but it is minuscule compared to the time it would take to provision, secure, and manage a group of containers for an open source project.
sambe 10y ago
Do you really find Travis easy to use and configure? I've used TeamCity a lot and find the functionality to be decent to good but the UI horrible. Perhaps a bit biased because I had access to the server and agents in that case. A lot of debugging/discoverability issues are easier with full access of course. I've then used Travis a bit for small GitHub projects (and contributing to other people's projects) and it has a nice UI for simple stuff. However, it gives the impression that they are struggling to stay alive - many features are in beta or feel like they are; a lot is under-documented or planned soon (for multiple years). I know I can dig into their various projects' source code to find out how things work, but I always feel incredibly unproductive figuring something out in Travis. I worry for them. Great to hear you have a good experience with GitLab!

sandGorgon 10y ago

how are you deploying secrets to your kuberenetes CI pipeline ? for example on a build, you create (I'm assuming) a new namespace. it seems you are using glusterfs for your persistentvolumeclaim.. why not gce-pd ?

tzaman 10y ago
Juenkins' Kubernetes plugin allows to mount secrets like with usual pods so we deploy test secrets just like we normally do production ones. As for glusterfs, our Nginx reverse proxy also has a cron job that runs letsencrypt automatic renewal once a week so it needs to be able to write those new certificates. Because we need to be able to run several reverse proxies, all with write persmissions (we run one at a time, but on deployment, there are two running to avoid downtime) we chose gluster, as gce-pd doesn't support multiple writers.
- sandGorgon 10y ago
  Oh wow. I have never used glusterfs on GCE - how difficult was it to set up and manage it ? What about ceph ?

Kudos 10y ago

FWIW, Circle CI offers the ability to SSH in to build nodes and free nodes for public projects

fredley 10y ago
It's really well implemented too, it grabs authorised public keys from the github repo directly, so if you can push to the repo for the project, you can just ssh straight into the build machine. Magic.
gurgus 10y ago
Was looking for this comment! CircleCI is amazing. We use it and it seems to offer us everything we need in a very nice shiny box.
haldean 10y ago
...sort of. I eventually rage-quit Circle and set up a Jenkins cluster because of all of the heisenbugs we found on Circle. Builds would fail 5-10% of the time for totally unreproducible reasons (for example, pip install into a venv would fail with a permission error), and you can't SSH into a build that's already failed. We very rarely had problems with Jenkins builds, and when we did, we could go look at the environment it had run in and diagnose what went wrong. I love Jenkins and would absolutely choose it over a hosted solution. (We also went from paying $1k/mo to $0/mo, which is a very nice side effect)
- tzaman 10y ago
  > (We also went from paying $1k/mo to $0/mo, which is a very nice side effect) That's not entirely true, because someone had to set Jenkins up and has to maintain it, but once things are rolling, it hardly needs any input.
- pm90 10y ago
  Agree completely. I think new users are perhaps turned off by Jenkins due to its (deserved) reputation for ugly UI. But its very dependable, stable and reliable solution. And there are plugins for doing every conceivable thing. We have a job manager to store the config in CI, and its such a breeze to work with.

scrollaway 10y ago

The problem with travis and the like is most do not support arbitrary builds. Drone for example is trying to sell itself as a "jenkins replacement" but has no concept of a build triggering arbitrarily, or that isn't intrinsically linked to a git repo. It's nonsense. Once you set up a bunch of tooling jobs on Jenkins it's very nice to be able to use it as some form of control center for a bunch of different operations. Calling it a build server is underselling it.

sytse 10y ago
I think there are two aspects to arbitrary builds: 1. Periodic jobs (not linked to a pushed commit) 2. Not linked to a git repo At GitLab we try to have 'infrastructure as code'. We think builds should be under version control to allow for collaboration. But periodic jobs make a lot of sense and it is discussed in https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 https://gitlab.com/gitlab-org/gitlab-ce/issues/2989
- jacques_chester 10y ago
  I think one thing GitlabCI lost in taking inspiration from Concourse (amongst other tools!) was the centrality of resources. It wasn't obvious at first why this is so important, I and most others fixated on the other visible differences (containers for everything, declarative config, pipeline view etc). There are a lot of features that don't need to be added to the core because they can be resources instead. For git-triggered builds, I use the git resource. Periodic, I use the time resource or a cron resource. S3 triggered, I use the S3 resource. Triggered on release of new software to PivNet, I use pivnet-resource. Triggered on a new docker image, I use docker image resource. And so on. Any resource that has a 'get' operation can trigger a build. 3rd parties can add triggers without needing to avoid interference with other resources, because every operation is isolated. Disclosure: I work for Pivotal, which sponsors Concourse.
  - sytse 10y ago
    Concourse is great and it was certainly an inspiration for GitLab CI. What I liked most and what we're still working on are cross project triggers https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 The use of resources sounds cool.

arekkas 10y ago

Are you running test code in production environment? Not sure about this one

tzaman 10y ago
I wasn't at first either, but benefits outweigh the risks (which are all security related). It's something I'm willing to re-think if it turns out to be problematic on any level. But so far after half a year in production, we've seen no problems.

donatj 10y ago

We run Drone and are very happy with it. Really easy to install and get working, similar but better usage than Travis, open source, Docker powered. I really hope it catches on more strongly because it's fantastic.

chrisabrams 10y ago

We are doing the exact same thing and also couldn't be happier :)