The physical Sackur-Tetrode entropy of an ideal gas

Marton Trencseni - Mon 29 November 2021 • Tagged with entropy, physics

I derive the Sackur-Tetrode equation for entropy of a monatomic ideal gas.

Continue reading

Entropy of an ideal gas with coarse-graining

Marton Trencseni - Fri 19 November 2021 • Tagged with entropy, physics

I show the first steps of how to arrive at a definition of entropy for a monatomic ideal gas modeled as hard billiard balls.

Continue reading

Entropy in Data Science

Marton Trencseni - Sun 24 October 2021 • Tagged with entropy

I discuss 4 uses of entropy in Data Science: (i) cross entropy as a loss function for training neural network classifiers (ii) entropy as a splitting criterion for building decision trees (iii) entropy for evaluating clustering algorithms (iv) entropy for understanding relationships in tabular data.

Information gain

Continue reading

Cross entropy, joint entropy, conditional entropy and relative entropy

Marton Trencseni - Sat 09 October 2021 • Tagged with entropy, cross-entropy, joint-entropy, conditional-entropy, relative-entropy, kullbackÔÇôleibler-diverence

What's the difference between cross entropy, joint entropy, conditional entropy and relative entropy?

Information gain

Continue reading

What's the entropy of a fair coin toss?

Marton Trencseni - Sat 25 September 2021 • Tagged with entropy, interviews, cross-entropy, physics

What's the entropy of a fair coin toss? What if the coin almost always returns Heads? My recruiter reports that very few candidates can answer these entropy related DS screening questions.

Entropy

Continue reading