← Back

Entropy

Engineering Degree Guide: Studying Entropy for Chemical Engineering

The term “entropy” was created by a German scientist, Rudolf Clausius, to describe the measure of energy wasted by an isolated system (in other words, the energy not used to perform work). Energy loss occurs in virtually all systems, as complete efficiency is hard to accomplish. Entropy answers many questions important to physics, such as why processes tend to progress spontaneously in one form but not another. The idea of entropy is applied to statistical models in physics to try to determine the distribution of particles in a given system.

Almost a century later, scientist Claude Shannon theorized a version of entropy to serve information theory. His work on coding languages and compression defines entropy slightly differently than his forerunners. Entropy, for information science, describes the ability to predict the next character in a series of random variables. Shannon applied many of the concepts of thermodynamic entropy to his informational reappraisal including efficiency, additive properties and the assertion that energy only increases through system state transformations. In fact the formulas for determining thermodynamic and informational entropy are almost identical.

Shannon’s work on probabilistic entropy inspired mathematicians to take up the cause as well. In math entropy can refer to a few different measurements of the complexity of a system. The specifics of mathematic entropy are caught up in complex formulas and concepts, but they generally resemble previous formulas describing entropy. With the flurry of research surrounding entropy, researchers in the social sciences began using the term as a way to describe certain tendencies in their field. Entropy has been used as a way to describe urban decay, voting tendencies and even political campaigns. Informational sciences may have more to offer humanities in the fields of linguistics and literature. The use of the term entropy in social sciences, however, is not well defined and often describes completely different phenomena based on the researcher and topic studied. For this reason most scientists view applications of entropy as a less structured concept with skepticism.

Depending on the application of the term, disorder can mean many things, but in general it denotes approaching a state of equilibrium. For entropy, order is the function of external forces acting on a system and disorder is the tendency of systems to resist these types of changes. Research into entropy has helped us achieve unseen productivity in business sectors and has promoted research into systems too complex to be broached before. It is an exciting area of study, yet at the same time brings frightening questions such as the possibility of heat-death, or a state in which matter can no longer perform work. Despite these fears, the research conducted into the nature and applications of entropy has helped us understand our world and inspired discussions in science, art, religion, and biology. To help start your own exploration of entropy, listed below is a related compilation of resources.

 

Informational Entropy

  • Typical Sequences and All That: Entropy, Pattern Matching and Data Compression – Aaron D. Wyner’s lecture on entropic systems in computing and information theory. Wyner’s work sets out mathematical formulas to address common informational problems to classify their entropy.
  • A Short Course in Information Theory - Lecture notes from an information theory class taught by professor David MacKay. The first lecture is especially focused on informational entropy and its different manifestations in data.
  • Achieving the Shannon Limit: A Progress Report – A report on research into maximizing the use of channel capacity and the effects of entropy on doing so. This is from Robert McEliece lecture at the Thirty-Eighth Allerton Conference of 2005.

Systems

  • Symbolic Dynamics and Coding Applications – An investigation of binary systems and their tendency toward entropic states. Uses principles primarily found in coding to illustrate the uncertainty of outcomes in dynamic systems.
  • An Entropy Primer – Introduces concepts related to system dynamics and integrates these with information entropy theory. The text offers explanation of Claude Shannon’s Noiseless Coding and Equipartition Theorems.
  • All Entropies Agree for an SFT – A research paper aimed at developing relationships between different conceptions of entropy. This type of relational approach is especially important for efforts in integration, such as String Field Theory.

Mathematics and Algorithms

  • Lecture Notes on Descriptional Complexity and Randomness - Provides algorithmic examples of entropy with applications to real world applications. The paper stresses the importance of entropy to understanding probability and complex algorithmic equations.
  • Randomness and Mathematical Proof – Discusses conceptions of randomness as examples of entropic systems. This paper provides many examples to help the beginning reader understand complicated concepts.
  • Jonathan Borwein - A mathematics professor at New Castle University. His papers and books contain great information on mathematic entropy and information theory.
  • Randomness in Arithmetic - An explanation of how concepts which we think of as stable, such as arithmetic operations, actually exhibit a high degree of randomness. The paper gives interesting examples of the usefulness of this theory to problems in biology and beyond.
  • The Discovery of Algorithmic Probability – Describes the background and development of statistical representation in algorithms. This file is in PostScript format and you will need the appropriate program to read it.
  • Prediction and Information Theory - Describes the importance of prediction in constructing algorithmic equations. This paper goes into detail on many different types of prediction and the problems inherent to each.
  • Probability Theory: The Logic of Science – An expository book on the nature of statistical probability and its uses to deduce possible outcomes. The book is downloadable with a PostScript file for each chapter.

Economic Entropy

  • Institute of Business Entropy – A research facility dedicated to developing models of entropy applicable in business situations. The institute publishes papers on entropy and business and consults with business and government entities.

Thermodynamic Entropy

  • A Brief Maxent Tutorial - A description of the importance of maximal entropy to statistical modeling and probability. The paper focuses on the application of entropic models to natural language processing but maintains their validity in a variety of situations.
  • Information and Correlation in Statistical Mechanical Systems - A detailed research paper on the how we gain and interpret information in scientific processes. It describes how entropy affects seemingly ordered processes and distorts information.
  • The Black Hole Information Loss Problem – A discussion of Hawking radiation and how a black hole transforms physical objects made up of many variables into a change in temperature. This finding presents problems for physics since it posits that information, the description of the physical state of the object, is lost through the process.

Biological Entropy

Cultural

  • Wikipedia: Social Entropy – A basic outline of how entropy relates to sociology and cultural structures. Since entropy’s application to social structures is a relatively new development, this page gives some good ideas on where it has the most to offer.
  • A Test for Conformity in Voting Behavior - An interesting paper designed to test whether voting tendencies subscribe to a probabilistic system. The document does a great job of explaining informational entropy and separating it from physical descriptions of uncertainty.

Research

  • Entropy - An open access journal presenting studies on entropy and related topics. The journal is peer reviewed and incorporates research from many disciplines.
  • Journal of Information Science – A subscription based peer-reviewed journal focused in the field of information theory. Not all topics relate to entropy, but it is a popular and important focus of current studies of information systems.
  • American Physical Society Journals – The American Physical Society publishes many journals on various topics important to the physical sciences. Almost all of these publications have touched on entropy at some point.
  • Journal of the American Chemical Society - Although focused on the science of chemistry, this journal does delve into entropy related topics in biology. This journal is peer reviewed and contains articles as well as book and software reviews.
  • Caltech Communications Group – The site for communications studies for the California Institute of Technology. The site lists faculty and students of the program as well as links to outside sources related to communication.
  • The Information Systems Laboratory – Stanford’s research lab for inquiries into information systems. Their site provides an overview of research, available classes and news on information systems
  • Center for Advanced Studies – The University of New Mexico’s physics research center involved in many aspects of investigation. Information physics is one of their major focuses.

Entropy Events

  • Winter School in Complex Analysis and Geometry – A three course program for studies in how to handle complex systems and analyze geometric patterns. Also includes a workshop with guest speakers on dynamical systems.
  • Frontiers in Complex Dynamics: Celebrating John Milnor’s 80th Birthday – An investigation of holomorphic functions and their potential in solving some complex functions of mathematics. The conference is designed to put students in touch with experienced researchers to encourage discovery.
  • Workshop on Discrete Methods in Ergodic Theory – A series of speeches delivered by experts in ergodic theory and dynamical systems. The conference takes place at Northwestern University and includes the presentation of the Nemmers Prize in Mathematics.
  • Trends in Dynamics – A discussion of current and continuing research into systems with finite state possibilities. This conference may provide funding for graduate students.

Entropy Tools

  • The Natural Language Software Registry - A list and summary of programs dedicated to processing the ways humans understand and identify language. The applications are divided into categories based upon their approach to NLP and the site allows user submitted entries.