6 ms·

Ignition: A new CoreOS machine provisioning utility

67 points by polvi 10y ago

HorizonXP 10y ago
Alex this looks great and likely addresses a lot of the problems I had with cloud-config. I was one of those with a Bash script that I used to generate multiple files, so this is great. My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces. I've solved this by adding net.ifnames=0 to my grub.cfg. It requires that I reboot the machine at least once to get it to take. If I could have predictable interface names using Ignition, then I'm set!
- jsmthrowaway 10y ago
  > My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces. That's a systemd decision, not CoreOS, and also impacts the 7.x series of RHEL derivatives and anything that uses systemd > v197, really. Your way is one of three to revert it. https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ https://www.freedesktop.org/wiki/Software/systemd/Predictabl...
  - HorizonXP 10y ago
    Ah, fair enough. But it looks like that introduces another problem. When you spin up a VM in ESXi, it generates a new MAC address for each interface. So it would be great to create systemd .network files that bind to those MAC addresses. Except you don't what MAC addresses you need to match against. The only flow I see working is: 1) Create a VM with your interfaces, but don't boot it. 2) Take the MAC addresses and enter them into your .network files, created via cloud-config/Ignition 3) Mount the cloud-config/Ignition file to your VM and boot Which is a bit painful to do manually when you have 10 VMs, with 5-10 interfaces each. I'd love to automate this, and if you happen to have a suggestion here, I'd really appreciate it. Edit: Oh, and when I said it had an eno* number, it's more like eno16777736, which is not very predictable at all.
    - wmf 10y ago
      You don't have to match against MAC addresses, although there's virtually no documentation about how to do it any other way. In the pre-systemd past I've successfully created udev rules that match PCI slots so that the configuration does not depend on MAC addresses. It looks like the same thing should be possible (with trial and error) with path globbing in systemd.
- justinsaccount 10y ago
  Are you talking about things like eno16777736 ? it's a vmware issue... http://serverfault.com/questions/636621/why-is-my-eth0-called-eno16777736 http://serverfault.com/questions/636621/why-is-my-eth0-calle...
  - HorizonXP 10y ago
    Yes, precisely. The comment below identified one of 3 possible routes to turn off the naming, one of which I'm using (net.ifnames=0 in the kernel config). If I could find a better way to match the interfaces, it would be fantastic.
- otterley 10y ago
  Why do you need predictable interface names if you're using DHCP?
  - HorizonXP 10y ago
    I am not using DHCP.
    - otterley 10y ago
      You should; it'll solve your problems.
      - seanp2k2 10y ago
        Yes, use DNS for everything, then you can use IPv6 too, without having to remember those pesky long addresses!
brandonbloom 10y ago
Apologies for the somewhat off-topic remark, but ... Can we please stop creating new JSON-based config files? Sane config file formats must support comments.
- kkirsche 10y ago
  100% agreed. This is the reason I personally find YAML to be a great config alternative to JSON due to their ease of converting between the two
  - brandonbloom 10y ago
    YAML isn't a particularly good config format either, since it has a lot of unnecessary complexity. For example, bare words are strings, but some bare words, such as "true" are booleans. Which wouldn't be so bad, but did you know that "yes" and "no" are also booleans? I've gotten YAML files from an i18n-system that screwed this up in a pretty bad way.
  - jonaf 10y ago
    In fact, YAML is a superset of JSON, and any valid JSON is also valid YAML. This becomes more obvious when you realize you can put any JSON objects inside of your YAML document and any YAML parser will correctly deserialize it.
- Sanddancer 10y ago
  I wouldn't apologize for that, you're addressing the elephant in the room here. JSON has turned into the new XML. There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good. So it seems to be used quite a bit in lieu of looking carefully at the problem and figuring out the best structured format. Configuration files tend to have a lot of ambiguity in them, going to a format that makes it all that much harder to pull out the why part of settings does nothing to improve on that.
  - geofft 10y ago
    > There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good. I don't get that from the article. There's a stated reason, that it's a stricter format with much less risk of type confusion, which sounds plausible enough to me. There's also the benefit that tool support is good. Why does the web need to be relevant at all?
darren0 10y ago
The transition away from cloud-init and YAML is really quite odd to me. Nobody enjoys editing JSON files. Forgetting commas, no multi-line strings, no comments, escaping characters in strings, etc. Just reading the documentation of ignition should be enough to illustrate the pain it is to manage multiline content for unit files in an JSON string. But why abandon the cloud-init format in general? Again, why would somebody want to learn a new configuration syntax. Using CoreOS already requires you to know and use systemd units (most other distros this really isn't required knowledge), so that adds two steps to users learning/using CoreOS.
- nikolay 10y ago
  YAML is good for simple configs. As the article says, JSON is much easier to programmatically generate than non-standard YAML. You still can store files in YAML, but compose and convert them into JSON in user data.
  - deleted 10y ago
    [deleted]
  - seanp2k2 10y ago
    Again here, now you have 2 problems. Why not just go with a simpler config language and skip converting it and all the complexity that adds? It's possible to make this extremely complex, to the point where the config is in a database with ACLs and requires a DB server system (which itself requires config) to host (e.g. WebLogic) , but what do you gain?
- chrissnell 10y ago
  I'm totally with you. We're big users of CoreOS and while there are aspects of cloud-init that I find janky, it's at least mostly easy to read and understand. The use of JSON is nuts. Just looking at the one-shots consolidated to a single line with \n's gives me a headache. This is moving forwards? What I would prefer is a compiler. You feed it a directory of unit file drop-ins and app config templates and it builds a single artifact that can be served over HTTP and pulled by the server booting up. This could allow for dynamic configuration and automation but still makes it easy for the admin to piece the config together.
rdtsc 10y ago
> Running before systemd helps ensure that all services established by Ignition are known to and can be managed by systemd when it subsequently starts. This allows systemd to do what it does best: concurrently start services as quickly as possible. I think this looks like a slight jab at systemd ;-) Systemd thinks it does a lot more other things best
- wmf 10y ago
  I think that was poorly explained on their part. Digging deeper into the docs, it looks like Ignition writes unit files that systemd runs; e.g. networkd is still used to configure networking.
  - thebeardisred 10y ago
    Affirmative. This is just a mechanism to make sure that interfaces can be properly plumbed, disks formatted with the desired states & filesystems, etc.
songgao 10y ago
I'm curious about how it works. I must be missing something, but if it runs before the network and file system are setup, how does it pull configure from a http:// http:// URL (say on bare metal) or oem:// which is on FS? Does it run with a default setup for network/fs and let systemd redo everything later?
fpoling 10y ago
JSON is not language that "makes it very easy to write tools to generate new configs or manipulate existing ones". One cannot realistically consume JSON in a shell script, and even with tools that have native JSON support manipulating the config require the knowledge of document semantics as the language itself does not define how to combine or merge 2 documents into one. For example, consider an application that comes with a complex JSON (or YAML as that language also share this shortcomings) that essentially describes default settings. I cannot just define an extra file that specifies a couple settings that I alter/delete/add, I have to have my own copy of the original file with my changes resulting in painful merge when the original is updated. What I have found is that things like .ini files or config files in style of ssh_config work much better in practice. It is easy to generate and process them in any language with a notion of text IO including shell scripts and the merging functionality can be provided independent of semantics so I can keep config fragment outside the main file/files.
dghubble 10y ago
Some useful Ignition configs can be found in the https://github.com/coreos/coreos-baremetal https://github.com/coreos/coreos-baremetal examples. For those wondering about the format, they're converted to JSON and served to machines.

67 points by polvi 10y ago

Alex this looks great and likely addresses a lot of the problems I had with cloud-config. I was one of those with a Bash script that I used to generate multiple files, so this is great. My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces. I've solved this by adding net.ifnames=0 to my grub.cfg. It requires that I reboot the machine at least once to get it to take. If I could have predictable interface names using Ignition, then I'm set!

jsmthrowaway 10y ago
> My biggest issue so far is CoreOS' naming of Ethernet interfaces on VMWare ESXi. It always uses some eno* name for each interface. I have a unique case where each VM I spin up has up to 10 interfaces. That's a systemd decision, not CoreOS, and also impacts the 7.x series of RHEL derivatives and anything that uses systemd > v197, really. Your way is one of three to revert it. https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ https://www.freedesktop.org/wiki/Software/systemd/Predictabl...
- HorizonXP 10y ago
  Ah, fair enough. But it looks like that introduces another problem. When you spin up a VM in ESXi, it generates a new MAC address for each interface. So it would be great to create systemd .network files that bind to those MAC addresses. Except you don't what MAC addresses you need to match against. The only flow I see working is: 1) Create a VM with your interfaces, but don't boot it. 2) Take the MAC addresses and enter them into your .network files, created via cloud-config/Ignition 3) Mount the cloud-config/Ignition file to your VM and boot Which is a bit painful to do manually when you have 10 VMs, with 5-10 interfaces each. I'd love to automate this, and if you happen to have a suggestion here, I'd really appreciate it. Edit: Oh, and when I said it had an eno* number, it's more like eno16777736, which is not very predictable at all.
  - wmf 10y ago
    You don't have to match against MAC addresses, although there's virtually no documentation about how to do it any other way. In the pre-systemd past I've successfully created udev rules that match PCI slots so that the configuration does not depend on MAC addresses. It looks like the same thing should be possible (with trial and error) with path globbing in systemd.
justinsaccount 10y ago
Are you talking about things like eno16777736 ? it's a vmware issue... http://serverfault.com/questions/636621/why-is-my-eth0-called-eno16777736 http://serverfault.com/questions/636621/why-is-my-eth0-calle...
- HorizonXP 10y ago
  Yes, precisely. The comment below identified one of 3 possible routes to turn off the naming, one of which I'm using (net.ifnames=0 in the kernel config). If I could find a better way to match the interfaces, it would be fantastic.
otterley 10y ago
Why do you need predictable interface names if you're using DHCP?
- HorizonXP 10y ago
  I am not using DHCP.
  - otterley 10y ago
    You should; it'll solve your problems.
    - seanp2k2 10y ago
      Yes, use DNS for everything, then you can use IPv6 too, without having to remember those pesky long addresses!

brandonbloom 10y ago

Apologies for the somewhat off-topic remark, but ... Can we please stop creating new JSON-based config files? Sane config file formats must support comments.

kkirsche 10y ago
100% agreed. This is the reason I personally find YAML to be a great config alternative to JSON due to their ease of converting between the two
- brandonbloom 10y ago
  YAML isn't a particularly good config format either, since it has a lot of unnecessary complexity. For example, bare words are strings, but some bare words, such as "true" are booleans. Which wouldn't be so bad, but did you know that "yes" and "no" are also booleans? I've gotten YAML files from an i18n-system that screwed this up in a pretty bad way.
- jonaf 10y ago
  In fact, YAML is a superset of JSON, and any valid JSON is also valid YAML. This becomes more obvious when you realize you can put any JSON objects inside of your YAML document and any YAML parser will correctly deserialize it.
Sanddancer 10y ago
I wouldn't apologize for that, you're addressing the elephant in the room here. JSON has turned into the new XML. There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good. So it seems to be used quite a bit in lieu of looking carefully at the problem and figuring out the best structured format. Configuration files tend to have a lot of ambiguity in them, going to a format that makes it all that much harder to pull out the why part of settings does nothing to improve on that.
- geofft 10y ago
  > There seems to be a pervasive undercurrent that if it looks website-ish, it's got to be good. I don't get that from the article. There's a stated reason, that it's a stricter format with much less risk of type confusion, which sounds plausible enough to me. There's also the benefit that tool support is good. Why does the web need to be relevant at all?

darren0 10y ago

The transition away from cloud-init and YAML is really quite odd to me. Nobody enjoys editing JSON files. Forgetting commas, no multi-line strings, no comments, escaping characters in strings, etc. Just reading the documentation of ignition should be enough to illustrate the pain it is to manage multiline content for unit files in an JSON string. But why abandon the cloud-init format in general? Again, why would somebody want to learn a new configuration syntax. Using CoreOS already requires you to know and use systemd units (most other distros this really isn't required knowledge), so that adds two steps to users learning/using CoreOS.

nikolay 10y ago
YAML is good for simple configs. As the article says, JSON is much easier to programmatically generate than non-standard YAML. You still can store files in YAML, but compose and convert them into JSON in user data.
- deleted 10y ago
  [deleted]
- seanp2k2 10y ago
  Again here, now you have 2 problems. Why not just go with a simpler config language and skip converting it and all the complexity that adds? It's possible to make this extremely complex, to the point where the config is in a database with ACLs and requires a DB server system (which itself requires config) to host (e.g. WebLogic) , but what do you gain?
chrissnell 10y ago
I'm totally with you. We're big users of CoreOS and while there are aspects of cloud-init that I find janky, it's at least mostly easy to read and understand. The use of JSON is nuts. Just looking at the one-shots consolidated to a single line with \n's gives me a headache. This is moving forwards? What I would prefer is a compiler. You feed it a directory of unit file drop-ins and app config templates and it builds a single artifact that can be served over HTTP and pulled by the server booting up. This could allow for dynamic configuration and automation but still makes it easy for the admin to piece the config together.

rdtsc 10y ago

> Running before systemd helps ensure that all services established by Ignition are known to and can be managed by systemd when it subsequently starts. This allows systemd to do what it does best: concurrently start services as quickly as possible. I think this looks like a slight jab at systemd ;-) Systemd thinks it does a lot more other things best

wmf 10y ago
I think that was poorly explained on their part. Digging deeper into the docs, it looks like Ignition writes unit files that systemd runs; e.g. networkd is still used to configure networking.
- thebeardisred 10y ago
  Affirmative. This is just a mechanism to make sure that interfaces can be properly plumbed, disks formatted with the desired states & filesystems, etc.

songgao 10y ago

I'm curious about how it works. I must be missing something, but if it runs before the network and file system are setup, how does it pull configure from a http:// http:// URL (say on bare metal) or oem:// which is on FS? Does it run with a default setup for network/fs and let systemd redo everything later?

fpoling 10y ago

JSON is not language that "makes it very easy to write tools to generate new configs or manipulate existing ones". One cannot realistically consume JSON in a shell script, and even with tools that have native JSON support manipulating the config require the knowledge of document semantics as the language itself does not define how to combine or merge 2 documents into one. For example, consider an application that comes with a complex JSON (or YAML as that language also share this shortcomings) that essentially describes default settings. I cannot just define an extra file that specifies a couple settings that I alter/delete/add, I have to have my own copy of the original file with my changes resulting in painful merge when the original is updated. What I have found is that things like .ini files or config files in style of ssh_config work much better in practice. It is easy to generate and process them in any language with a notion of text IO including shell scripts and the merging functionality can be provided independent of semantics so I can keep config fragment outside the main file/files.

dghubble 10y ago

Some useful Ignition configs can be found in the https://github.com/coreos/coreos-baremetal https://github.com/coreos/coreos-baremetal examples. For those wondering about the format, they're converted to JSON and served to machines.