7 ms·
EOF is not a character
- schoen 7y agoRecently (though mine was the only comment): https://news.ycombinator.com/item?id=22461647 https://news.ycombinator.com/item?id=22461647
- nixpulvis 7y agoWell then try explaining ctrl+c vs ctrl+d to someone who's never touched a terminal at all. Starts off so easily... "see one tells the program to stop" the other, well, if you're in a shell... or some programs... oh god. IDK anymore, just assume it works. What was the question?"
- ChristianBundy 7y agoMaybe you can correct me if I'm wrong, but I've always considered Ctrl+C and Ctrl+D to be signals that you can send a process rather than explicit characters. You might also get some stdout for those key combinations because ???, but they should be thought of as signals rather than as characters you're sending via stdin. Hoping Cunningham's Law comes into play with this comment. :)
- anonymousiam 7y agoControl-C is part of POSIX job control. If a stream (or "cooked" tty) sends a control-C (ASCII End-Of-Text or ETX), the foreground process will be sent a SIGINT signal. If that signal is not handled, the default action is to terminate the process (SIGTERM). Control-D is just another control character and not part of POSIX job control, but in the "cooked" case above, it will be interpreted as EOF and the process doing the "read" will receive that.
- nixpulvis 7y agoI was thinking the same thing, until I read this: > 'stty -icanon' still interprets control characters such as Ctrl-C whereas 'stty raw' disables even this and is the real raw mode. From the very detailed link posted by rgoulter above. Still, in raw mode, Ctrl+D will send EOT, and thus end your shell. While Ctrl+C wont.
- deleted 7y ago[deleted]
- rgoulter 7y agoI liked this explanation. https://www.linusakesson.net/programming/tty/ https://www.linusakesson.net/programming/tty/ When the TTY device takes (by default) Ctrl+C or Ctrl+D, it sends the signals to the program. The TTY's 'line discipline' (the policy for when the program's STDIN can read from a line of input) can be changed from a default 'cooked' to a 'raw mode'. In with raw mode line discipline the Ctrl+C doesn't send the signal. Presumably that's why e.g. vi or emacs don't just close on Ctrl+C.
- nixpulvis 7y ago> Now you press ^Z. Since the line discipline has been configured to intercept this character (^Z is a single byte, with ASCII code 26), you don't have to wait for the editor to complete its task and start reading from the TTY device. Instead, the line discipline subsystem instantly sends SIGTSTP to the foreground process group. This helps me, thanks for pointing me back at this great write-up.
- taeric 7y agoThis actually doesn't seem that hard. In both, you are telling the computer, not the target program, something. One is to signal the running program you want to interrupt it. The other is to close the input to the program, as you are done giving it data.
- 1996 7y agoit all depends on your stty settings. since I am more used to Windows where ctrl-c is copy, I followed other people's suggestion and mapped ctrl-x to do what ctrl-c usually does, with: stty intr ^X -ixon This is because X and C are very close, and I couldn't sacrifice ctrl-v (paste) or ctrl-z (background) while I seldom use ctrl-c I'm sure you could do the same with ctrl-d if you really wanted to.
- pwdisswordfish2 7y agoYou could just use Ctrl-Insert/Shift-Insert for copy/paste everywhere.
- nixpulvis 7y agoI find it interesting that Rust's `Read` API for `read_to_end` [1] states that it "Read[s] all bytes until EOF in this source, placing them into buf", and stops on conditions of either `Ok(0)` or various kinds of `ErrorKind`s, including `UnexpectedEof`, which should probably never be the case. [1]: https://doc.rust-lang.org/std/io/trait.Read.html#method.read_to_end https://doc.rust-lang.org/std/io/trait.Read.html#method.read...
- comex 7y agoThe reason for that is that, for simplicity's sake, all of the I/O functions share the same error type. `UnexpectedEof` should never be returned from `read_to_end`, but it can be returned from `read_exact`.
- cesarb 7y agoThat's because `UnexpectedEof` is never returned from `read()`, it's only ever returned from `read_exact()`. In fact, `UnexpectedEof` didn't exist originally, it was added together with `read_exact()` to represent its unique error case (which is: `read()` returned end-of-file, but we still needed more bytes to completely fill the buffer). It's an error to return `UnexpectedEof` from any of the other methods of the `Read` trait, and since it's an error, it makes sense for `read_to_end()` to stop and propagate that error. (In fact, thinking better about it, there are some cases where `read()` could legitimately return `UnexpectedEof`, like when it's a wrapper for a compressed stream which has fixed-size fields, and that stream was truncated in the middle of one of these fields. It's clear that, in that case, `UnexpectedEof` is not an end-of-file for the wrapper; it should be treated as an I/O error.)
- reidacdc 7y agoSeems like the confusion arises because getchar() (or its equivalent in langauges other than c) can produce an out-of-band result, EOF, which is not a character. Procedural programmers don't generally have a problem with this -- getchar() returns an int, after all, so of course it can return non-characters, and did you know that IEEE-754 floating point can represent a "negative zero" that you can use for an error code in functions that return float or double? Functional programmers worry about this much more, and I got a bit of an education a couple of years ago when I dabbled in Haskell, where I engaged with the issue of what to do when a nominally-pure function gets an error. I'm not sure I really got it, but I started thinking a lot more clearly about some programming concepts.
- nixpulvis 7y agoWhat does "Procedural" vs "Functional" have to do with this? It's a choice in data type. If by procedural you mean, nonsense, then sure... I agree that a function named `getchar` returning an `int` is procedural. :P
- chrisseaton 7y ago> If by procedural you mean, nonsense, then sure Why are you being snarky? They clearly mean the issue of modelling partial functions which would normally be done by a side-effect in a procedural language but can’t in a functional language.
- nixpulvis 7y agoNo, they imply that the handling is done by returning a negative number. I'm being snarky, as is my nature, to highlight the madness of a function called `getchar` returning anything but a `char`.
- samatman 7y agoIt's not a great snark given that the C standard considers the signedness of char to be implementation defined, making -1 a valid option, sometimes.
- deleted 7y ago[deleted]
- chrisseaton 7y agoSo what is CP/M-style character 26? Isn’t that documented as end-of-file?
- nixpulvis 7y agoI'm just reading up on this now. But according to Wikipedia "CP/M used the 7-bit ASCII set", so then character 26 would be the "SUB (substitute)" character. No? EDIT: Seems like 26 = EOF is a DOS thing. EDIT 2: Some confusing comments: https://www.perlmonks.org/bare/?node_id=228760 https://www.perlmonks.org/bare/?node_id=228760 EDIT 3: A pretty good thread (read NigelQ's replay): http://forums.codeguru.com/showthread.php?181171-End-of-File-Character http://forums.codeguru.com/showthread.php?181171-End-of-File...
- jcrawfordor 7y agoPerhaps a marginally better title would be "EOF is not a character [on Unix]". There are some OS that have an explicit EOF character, but it seems to have been the less common approach historically. CP/M featured an explicit end of file marker because the file system didn't bother to handle the problem of files which were not block-aligned, so the application layer needed to detect where the actual end of the file was located (lest it read the contents of the rest of the block). This is a pretty unusual thing to do, and was definitely a hassle for developers, so CP/M descendants like MS-DOS fixed it.
- kevin_thibedeau 7y agoThis also afflicts the xmodem protocol.
- mark-r 7y agoI think CP/M copied that convention from an even older OS but I can't remember which one.
- jcrawfordor 7y agoCP/M was developed on TOPS-10 and copied a lot of concepts from it. I can't immediately tell whether or not this is an example, but for any given eccentricity of CP/M it's a good bet that it came from TOPS-10. It's amusing that almost the same can be said about NT: for any given eccentricity of Windows NT it's a good bet that it came from VMS, since the two had the same principal designer.
- anonymousiam 7y agoCP/M and DOS use ^Z (0x1A) as an EOF indicator. More modern operating systems use the file length (if available). Unix/Linux will treat ^D (0x04) as EOF within a stream, but only if the source is "cooked" and not "raw". (^D is ASCII "End Of Transmission or EOT" so that seems appropriate, except in the world of unicode.)
- nixpulvis 7y agoUsing the "file length" as opposed to the "EOF indicator" is like how strings can either be represented as pointer to a contiguous sequence of `char` ending with a NULL byte, or as a tuple of (length, pointer), without the needed NULL byte. One gives a priori information the other a posteriori.
- schoen 7y agoStrictly speaking, as discussed elsewhere in this thread, ^D can cause a terminal device to signal an EOF condition; other kinds of Unix byte streams don't make this association. For example, $ python3 -c 'print("".join(chr(c) for c in range(10)))' | python3 -c 'print(list(ord(c) for c in input()))' will confirm that it doesn't happen in a pipe (the ASCII 4 character there is totally unrelated to EOF).
- pwdisswordfish2 7y agoThat is a common misconception. http://jdebp.info/FGA/dos-character-26-is-not-special.html http://jdebp.info/FGA/dos-character-26-is-not-special.html
- unilynx 7y agoI'm pretty sure the DOS TYPE command (its version of cat) would stop at the first ^Z it encountered, even if the file was longer. It was sometimes used to have TYPE print something human readable and stop before the remaining (binary) file data would scroll everything away
- cesarb 7y ago> It was sometimes used to have TYPE print something human readable and stop before the remaining (binary) file data would scroll everything away Notably, in the PNG file format (created back when MS-DOS was still very relevant): "The first eight bytes of a PNG file always contain the following values: [...] The control-Z character stops file display under MS-DOS. [...]" (http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R....)
- 1996 7y ago\r \n (0x0a 0x0d, or just one of them, or the combination of them, depending on your OS) is EOL ^D (0x04) is EOT and 0x03 is EOText: https://www.systutorials.com/ascii-table-and-ascii-code/ https://www.systutorials.com/ascii-table-and-ascii-code/ So, kinda, but somehow I'm happy it never got turned into a weird combinations depending on the OS.
- rectang 7y agoLike NULL, confusion over EOF is a problem which can be eliminated via algebraic types. What if instead of a char, getchar() returned an Option<char>? Then you can pattern match, something like this Rust/C mashup: match getchar() { Some(c) => putchar(c), None => break, } Magical sentinels crammed into return values — like EOF returned by getchar() or -1 returned by ftell() or NULL returned by malloc() — are one of C's drawbacks.
- nixpulvis 7y agoSo `read`'s `Ok(0)` result, is akin to `getchar`'s `None` result here. A different API causes a little more to consider, but generally makes sense.
- Someone 7y ago”What if instead of a char, getchar() returned an Option<char>?” Getchar doesn’t return a char; it returns an int (https://en.cppreference.com/w/c/io/getchar https://en.cppreference.com/w/c/io/getchar). ⇒ if C didn’t do automatic conversions from int to char, we would have that (in a minimalistic sense) That wouldn’t work for ftell and malloc (and, in general, most of the calls that set errno), though.
- rectang 7y ago> Getchar doesn’t return a char; it returns an int Dammit, I knew that. Thank you for flagging my blunder; being precise is really important in this case. The Linux manpage better explains the return value of getchar: https://linux.die.net/man/3/getchar https://linux.die.net/man/3/getchar "fgetc(), getc() and getchar() return the character read as an unsigned char cast to an int or EOF on end of file or error." getchar() needs to return an object the width of an unsigned char, but all the values in that range are taken by possible character values. The return type had to be expanded to int in order to accommodate the sentinel. The alternative of using an algebraic type is superior because the end-of-stream condition has a different type (so to speak), and furthermore, the programmer has no choice but to deal with it because the character value comes wrapped inside an Option which must be stripped away before the character value can be used. Really, you also want the type system to express all possible error conditions as well, since getchar() returning EOF can mean either that end-of-file was reached or that some other error occurred! As someone who has written lots of C code and worked hard to account for all possibilities manually, I really appreciate it when the type system and APIs can express all possibilities and back me up.
- jes5199 7y agoyeah, this author doesn’t know the history. Unix I/O was defined in opposition to practices in other OSes, that no longer exist
- guerrilla 7y agoClearly, since they barely know the system they are talking about but could you elaborate instead of leaving it vague? Which systems?
- jes5199 7y agothere’s plenty of other comments that explain it, but, CP/M, VAX, teletypewriters, punch cards - all used in-band control characters rather than an external signal
- Thorrez 7y agoAnother weird thing is that sometimes you can read an EOF, then keep reading more real bytes. So EOF doesn't necessarily mean the permanent end.
- jwilk 7y agoThe EOF condition for stdio functions is supposed to be sticky, although glibc didn't implement it correctly until 2.28: https://sourceware.org/bugzilla/show_bug.cgi?id=1190 https://sourceware.org/bugzilla/show_bug.cgi?id=1190 https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003... > All stdio functions now treat end-of-file as a sticky condition. If you read from a file until EOF, and then the file is enlarged by another process, you must call clearerr or another function with the same effect (e.g. fseek, rewind) before you can read the additional data. This corrects a longstanding C99 conformance bug. It is most likely to affect programs that use stdio to read interactive input from a terminal.
- Thorrez 7y agoWow, very interesting! That sounds like a somewhat significant change, and I wonder how much stuff will be broken by it. Although interestingly somehow I'm still seeing the old behavior in Debian Buster with glibc 2.28 with python3. import sys while True: b = sys.stdin.read(1) print(repr(b)) With old glibc with both python2 and python3 the EOF isn't sticky (as expected). With 2.28 with python2 the EOF is sticky (like you said). With 2.28 with python3 it's not sticky for some reason.
- badrabbit 7y agoBanged my head against the wall once after trying to figure out why Ctrl+D generates some character in bash but I can't send that character in a pipe to simulate termination.
- kylek 7y agoFun fact, ctrl-v in bash sets "verbatim insert" mode for the next character, so you can type a ^D "character" by doing "ctrl-v ctrl-d".
- pwdisswordfish2 7y agoIt’s not bash, it’s the tty device driver. Applications can switch between the ‘cooked’ mode (which recognises it as EOF) and ‘raw’ mode (which passes it through) by performing some ioctl I don’t really want to look up right now.
- nixpulvis 7y agoI think I may still be banging my head on this one. It's just an ioctl difference between my pipe and my terminal's session, right?
- enriquto 7y ago> Banged my head against the wall once after trying to figure out why Ctrl+D generates some character in bash but I can't send that character in a pipe to simulate termination. Yes, you can. You just end your stream by closing the pipe.
- Animats 7y agoIn the beginning, there was the int. In K&R C, before function prototypes, all functions returned "int". ("float" and "double" were kludged in, without checking, at some point.) So the character I/O functions returned a 16-bit signed int. There was no way to return a byte, or a "char". That allowed room for out of band signals such as EOF. It's an artifact of that era. Along with "BREAK", which isn't a character either.
- bhaak 7y agoYou can still today declare a function without a return value like this: "a() { return 1; }". GCC only outputs a warning by default: "warning: return type defaults to ‘int’ [-Wimplicit-int]"
- combatentropy 7y agoThe kernel returns EOF "if k is the current file position and m is the size of a file, performing a read() when k >= m..." So, is the length of each file stored as an integer, along with the other metadata? This reminds me of how in JavaScript the length of an array is a property, instead of a function that counts it right then, like say in PHP. Apparently it works. I've never heard of a situation where the file size number did not match the actual file size, nor of a time when the JavaScript array length got messed up. But it seems fragile. File operations would need to be ACID-compliant, like database operations (and likewise do JavaScript array operations). It seems like you would have to guard against race conditions. Does anyone have a favorite resource that explains how such things are implemented safely?
- JdeBP 7y agoYou are not thinking about it clearly. Ask yourself this: Filesystem formats use blocking and deblocking. How would a filesystem know the file size without having metadata for it?
- ineedasername 7y agoThis strikes me as the sort of pedantic and "I'm witty" click bait that occasionally percolates upwards on HN, especially considering the specifics of "EOF" are very much contingent on operating context.
- charlysl 7y agoThis is very well explained in the classic book The UNIX Programming Environment, by Kernighan and Pike, in page 44: Programs retrieve the data in a file by a system call ... called read. Each time read is called, it returns the next part of a file ... read also says how many bytes of the file were returned, so end of file is assumed when a read says "zero bytes are being returned" ... Actually, it makes sense not to represent end of file by a special byte value, because, as we said earlier, the meaning of the bytes depends on the interpretation of the file. But all files must end, and since all files must be accessed through read, returning zero is an interpretation-independent way to represent the end of a file without introducing a new special character. Read what follows in the book if you want to understand Ctrl-D down cold.
- unnouinceput 7y agoFor me EOF is a boolean state. Either I am at the end of file (stream / memory mapped etc) or not. That's how I was taught when I started programming. Never occurred to me to think of it like a character.
- IndexPointer 7y agoOf course it isn't, you couldn't have arbitrary binary files if one of the 256 possible bytes was reserved. That's why getchar returns int and not char; one char wouldn't be enough for 257 possible values (256 possible char values + eof).
- cjohansson 7y agoInteresting read, I suspected it was like this but I didn’t know for sure
- agumonkey 7y agoAnd this is why I failed C IO classes. Lack of information and improper abstraction.
- jwilk 7y agoUm, no, you can't use Python to infer that "EOF (as seen in C programs) is not a character". The exception even tells you that "chr() arg not in range(0x110000)" which has nothing to do with range of C's character types.