6 ms·

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thi

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, and entropy will help you quantify how many different ways that could have come to pass. e.g. is it a fair coin, a weighted coin, a coin with two head faces, etc. All these possibilities increase the entropy of the system. An arrangement _not_ counted towards the system's entropy is the arrangement where the coin has no heads face, only ever comes up tails, etc. Related, my intuition about the observation that entropy tends to increase is that it's purely a result of more likely things happening more often on average. Would be delighted if anyone wanted to correct either of these intuitions.

fsckboy 1y ago
>purely a result of more likely things happening more often on average according to your wording, no. if you have a perfect six sided die (or perfect two sided coin), none/neither of the outcomes are more likely at any point in time... yet something approximating entropy occurs after many repeated trials. what's expected to happen is the average thing even though it's never the most likely thing to happen. you want to look at how repeated re-convolution of a function with itself always converges on the same gaussian function, no matter the shape of the starting function is (as long as it's not some pathological case, such as an impulse function... but even then, consider the convolution of the impulse function with the gaussian)
russdill 1y ago
This is based on entropy being closely tied to your knowledge of the system. It's one of many useful definitions of entropy.
867-5309 1y ago
> 'actually be' and yet still 'appear to be' esse quam videri
tshaddox 1y ago
> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an observer come into play? It seems to me that two things in this scenario are true: 1) The fundamental physical interactions (i.e. particle collisions) are all time-reversible, and no observer of any one such interaction would be able to tell which directly time is flowing. 2) The states of the overall system are not time-reversible.
- thawtxp 1y ago
  [dead]
- dynm 1y ago
  It's tricky when you think of a continuous system because the "differential entropy" is different (and more subtle) than the "entropy". Even if a system is time-reversible, the "measure" of a set of states can change. For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed, the differential entropy changes. This is the kind of thing that happens in time-reversible continuous systems that can't happen in time-reversible discrete systems.
  - jdhwosnhw 1y ago
    I have yet to see differential entropy used successfully (beyond its explicitly constructed-for purpose for calculating channel capacity). Similar to your thought experiment is the issue that the differential entropy value depends on your choice of unit system. Fundamentally the issue is that you cant stick a quantity with units into a transcendental function and get meaningful results out
    - dynm 1y ago
      Yeah, it's quite disturbing that the differential entropy (unlike the discrete entropy) depends on the units. Even worse, the differential entropy can be negative! Interestingly, the differential KL-divergence (differential cross-entropy - differential entropy) doesn't seem to have any of these problems.

fsckboy 1y ago

>purely a result of more likely things happening more often on average according to your wording, no. if you have a perfect six sided die (or perfect two sided coin), none/neither of the outcomes are more likely at any point in time... yet something approximating entropy occurs after many repeated trials. what's expected to happen is the average thing even though it's never the most likely thing to happen. you want to look at how repeated re-convolution of a function with itself always converges on the same gaussian function, no matter the shape of the starting function is (as long as it's not some pathological case, such as an impulse function... but even then, consider the convolution of the impulse function with the gaussian)

russdill 1y ago

This is based on entropy being closely tied to your knowledge of the system. It's one of many useful definitions of entropy.

867-5309 1y ago

> 'actually be' and yet still 'appear to be' esse quam videri

tshaddox 1y ago

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an observer come into play? It seems to me that two things in this scenario are true: 1) The fundamental physical interactions (i.e. particle collisions) are all time-reversible, and no observer of any one such interaction would be able to tell which directly time is flowing. 2) The states of the overall system are not time-reversible.

thawtxp 1y ago
[dead]
dynm 1y ago
It's tricky when you think of a continuous system because the "differential entropy" is different (and more subtle) than the "entropy". Even if a system is time-reversible, the "measure" of a set of states can change. For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed, the differential entropy changes. This is the kind of thing that happens in time-reversible continuous systems that can't happen in time-reversible discrete systems.
- jdhwosnhw 1y ago
  I have yet to see differential entropy used successfully (beyond its explicitly constructed-for purpose for calculating channel capacity). Similar to your thought experiment is the issue that the differential entropy value depends on your choice of unit system. Fundamentally the issue is that you cant stick a quantity with units into a transcendental function and get meaningful results out
  - dynm 1y ago
    Yeah, it's quite disturbing that the differential entropy (unlike the discrete entropy) depends on the units. Even worse, the differential entropy can be negative! Interestingly, the differential KL-divergence (differential cross-entropy - differential entropy) doesn't seem to have any of these problems.