6 ms·
People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating
by blueskin_ 11y ago
People don't want it because it's binary, not because you can't grep it.
* you need to use a new proprietary tool to interact with them
* all scripts relating to logs are now broken
* binary logs are easy to corrupt, e.g. if they didn't get closed properly.
>You can have a binary index and text logs too! / You can. But what's the point?
The point is having human-readable logs without having to use a proprietary piece of crap to read them. A binary index would actually be a perfect solution - if you're worried about the extra space readable logs take, just .gz/.bz2 them; on decent hardware, the performance penalty for reading is almost nonexistent.
If you generate 100GB/day, you should be feeding them into logstash and using elasticsearch to go through them (or use splunk if $money > $sense), not keeping them as files. Grepping logs can't do all the stuff the author wants anyway, but existing tools can, that are compatible with rsyslog, meaning there is no need for the monstrosity that is systemd.
- oblio 11y agoWhat's wrong with Splunk? Honest question.
- fletchowns 11y agoIt's expensive
- blueskin_ 11y agoPrice, mostly. It's good, but there are alternatives that aren't as ridiculously expensive.
- oblio 11y agoAny alternative that you'd recommend? Thanks.
- linuxydave 11y agoELK stack - Elasticsearch, Logstash, and Kibana. The whole stack is opensource :)
- oblio 11y agoInteresting, but it's not a SaaS. It doesn't look like a direct rival to Splunk.
- linuxydave 11y agoIt is a direct rival to Splunk :) They do very similar things however IMHO Splunk is the better solution right now. There are LaaS companies that use ELK if you need a cloud solution - Loggly is the first one that springs to mind and I think another is LogSene.
- leni536 11y agoWhy does it have to be proprietary?
- tljr 11y agoDoesn't have to be but usually is.
- cthalupa 11y agoIt doesn't have to be - but let's look at reality here. NIH syndrome is everywhere, we have millions of competing protocols and formats, everyone thinks they can build a better solution than someone else, etc. I suppose that if there was a large push to universally log things in binary the possibility exists that sanity would prevail and we'd get one format that everyone agreed upon, but I don't see any reason that this would be the case when historically it basically never happens. So, at least from my prediction of a future where binary logging is the norm, we have a half dozen or so competing primary formats, and then random edge cases where people have rolled their own, all with different tools needed to parse them. Or we could stick with good ol' regular text files and if you want to make it binary or throw it in ELK/splunk or log4j or pipe it over netcat across an ISDN line to a server you hid with a solar panel and a satellite phone in Angkor Wat upon which you apply ROT13 and copy it to 5.25 floppy, you can do it on your own and inflict whatever madness you want while leaving me out of it.
- leni536 11y agoIt's not like text formats are universal. Thankfully we have settled on utf-8. The same could happen to a slightly more structured universal binary format that would be more suitable for many applications (like logging) and would have an established toolset just like 'text' now.
- bad_user 11y agoThat doesn't make sense. There can be no such thing as a universal binary format, unless you reinvent text/plain. Analyze a little on why we refer to some formats as being text-based and you'll see it.
- madhouse 11y ago* Why would you need a proprietary tool? * What if they get broken? I don't want to look at them raw anyway. * Text logs are easy to corrupt as well. Oh, append only? Well, you can do that with binary storage too. And again, there is no need for proprietary tools at all. Everything I want to do is achievable with free software - so much so, that I use only such software in all my systems. As for compressing - yeah, no. Please try compressing 100Gb of data and tell me the performance cost is nonexistent. As for LogStash & ES: Guess what: their storage is binary. Also note that my article explicitly said that the Journal is unfit for my use cases.