“You should never be surprised by or feel the need to explain why any physical system is in a high entropy state.”
― Brain Greene, The Fabric of the Cosmos

Adhyayam – 2

In the previous part, i gave a brief introduction about Artificial Intelligence, and Where do we stand in it currently.

Heuristics are techniques defined for a particular problem for solving it. They are a major part of algorithms that instruct machines to solve problems and make decisions.

Entropy is the name people came up with to describe why nothing is 100% perfect or 100% efficient. Technically, Entropy is the number of ways stuff can be arranged. And orderliness is not an universal definition, is it? What seems to be ordered to one can be disordered for another. so in my own formulation, I would define ordered system is one which has clear relationship among its various parameters or components. I can throw 10 red balls inside a box, and jiggle them. Every ball in that lot does not have the same relationship with other balls. There can’t be a perfect generic rule, whereas they can be arranged in a particular fashion so that there is some pattern followed if not a completely generic one. This we can all agree that is in a more ordered state than the previous. Now I pose a question – how many ways are there to achieve this pattern with the given system and how many messy ways?

It is reasonable to assume for a simplistic argument that each time we jiggle the box there is a new disordered arrangement of balls in the box and I can jiggle ‘n’ number of times every time coming up with a new arrangement, whereas we can arrange the balls in patterns only a certain number of times which is much less than ‘n’ . Thus more the number of ways of arrangement, higher is the entropy.

The common misconception is that Entropy is the measure of disorder but it is not. It is true that for most of the systems the number of disordered states is much higher that the number of ordered states, implying that higher the entropy higher the disordered state; leading to the above misconception. If so, we can argue this way If we make a system in which the number of ordered states are much higher that the number of disordered states, then the entropy will still be high implying a highly ordered state of the system. Yes, this is perfectly true and recently scientists have come up with such systems. Those are beyond the  scope of this post but I shall try to put up a post about them a little later.

After the brief description of entropy, we shall come back to AI. In most of the two-player games, the most favourable moves are the ones which has the maximum number of moves in the subsequent turn for that player compared to his opponent. If a move yields 10 different moves for your opponent and 15 moves for you in the next turn then it is a good move. Logically speaking it implies that there are more ways in which you can move than your opponent thus giving you the control of the game and a series of a controlled moves guarantees you a win. This is a general strategy followed by most of the two player algorithms for games. The most efficient one to me is the Alpha-Beta pruning algorithm which follows a similar logic. The general statement is ‘to maximise the diversity of future possible states’.

In the above example, the number of moves is one heuristic for evaluating the worthiness of a particular move. This is used in games like Othello, chess. There can be many such heuristics depending on the need and accuracy. Recently, people have used ‘Entropy’ as such a heuristic for making a system intelligent.

Having linked all the terms in the title, we shall proceed to the explanation. Alex Wissnergross proposed that entropy can be used as a heuristic. There is a law in Thermodynamics which I believe the most fundamental law in the whole of universe, which states that ‘Entropy of a  spontaneous process always increases in forward time’. Here, we need not worry about thermodynamic spontaneity but just get the idea. In our previous example, when we jiggle the box of balls, each event has more number of states to be arranged than the previous one. This is the implication of the Entropy Law.

Now we see the connection between Entropy and AI algorithms being, ‘Increase in the number of possible states’. That is what AI needs and that is exactly what Entropy provides. Simply Elegant!! Alex argues that Entropy is the “Underlying Mechanism of Intelligence”. He describes a force which drives systems to increase their entropy (called as entropic force) is the required heuristic of AI.

Entropic force

The above equation states that the force F seeks to maximize the possible diversity S with a strength T till a given time t (tau). Alex and his colleagues built a software engine called ‘Entropica’ which visualises and simulates the above equation in the given system. The main characteristic of Entropica is that only the system and the equation are fed in and the program by itself does intelligent actions without any goal or instruction, which is amazing and definitely the highest level of intelligence. In the video below, this is clearly demonstrated:

It is amazing how seemingly unrelated topics converge together and produce an astonishing result. If entropy can produce intelligent behaviour in systems on its own, is it also the reason behind human intelligence? This is an amazing question to investigate and might also provide answers why other animals are not as intelligent as humans.

Like, comment and share!!

Links & References:

Part 1 

Alex’s Page :  http://www.alexwg.org/

Entropica : http://www.entropica.com/

Alex’s TED talk : http://www.ted.com/talks/alex_wissner_gross_a_new_equation_for_intelligence




Entropy – The Next Heuristic for AI – Part 1

“Maybe the only significant difference between a really smart simulation and a human being was the noise they made when you punched them.”        ― Terry Pratchett, The Long Earth

Adhyayam – 1 [Chapter – 1 (Sanskrit)]

                                                       Artificial Intelligence is the future! It is an undeniable verity. Most of us would have heard or read about the annexation of Deep Mind Technologies , a mysterious AI company researching on Machine Learning by Google.inc early this year for $400 million. It was started in just 2011 and rose up to be one the most advanced AI companies albeit its clandestine work. So, there is an immense scope and need for AI.

What is AI?

I would state it this way : ” Any Algorithm or a procedure, which is capable of making rational decisions for various external stimuli. Well, that was short and precise wasn’t it? Just for a comparison, i will give you the widely accepted definition :

“The automation of activities that we associate with human thinking, activities such as decision making, problem solving, learning, …”  – Bellman, 1978.

Now the reader should observe that  in my definition, i was completely generic.Since the definition of AI by Bellman in 1978, there has been many fields introduced in AI such as visual perception, speech recognition, language processing, game playing. And notice the word automation. AI need not be an automation. We have loads of softwares that come under AI. The most important aspect of my definition is that i adhere to AI being capable of making only decision and that suffices. All others are just another way of saying the same. In game playing, you make decisions as to what the next optimal move should be, in language processing, you make decisions as to how to interpret the phrase and what is the suitable reply. In machine learning, you again choose the aspects that are to be incorporated in the next generation trials and their utility.  I also believe that the various other aspects of AI like Logic, reasoning, are also specific cases of decision making process.

Now the world has forged ahead from just pondering over the idea of the evolution of human intelligence to imparting it to machines. Each machine is a brain child of its creator. Now scientists try to assay the development of human intelligence through imparting intelligence to machines. Now a question arises, how close are the current AI algorithms to human cognition?, well not quite close but there has been tremendous furtherances in AI.

Okay.. so where are your fancy AI machines in the market?? Well, does Siri ring a bell? Google Now?. These are the products of natural language processing researches, one of the main domain in AI. AI is extremely vast and new kinds of intelligence are constantly budding up.

I would like to share some of the AI projects, which i feel are amazing and each is a work of art. I shall share 4 things which are jargon but everyone needs to know

  • Kilobots :  These are tiny mobile robots which are engineered to demonstrate collective behaviours algorithms. kilobot4Literally there are 1024 of them. These are useful for studying the swarm behaviour and implementing them in robotics for doing tedious tasks through intercommunicating bots. Developed by Self-Organizing systems research group at Harvard University. Their main aim is to control these 1000+ bots for various procedures like shape formation, contour tracing and localisation.
  • Wolfram Alpha : Some of you would have known about it but still, alphaThis is a computational knowledge engine. Well that’s how they call it. To put it in simpler terms, it is a search engine, which apart from from googling the answer (oops!) i meant searching the answer, it also computes the answer. Have a tough calculus problem? just put it in the wolfram alpha search bar and viola!! get the answer, the plot and various other relevant data! Compared to it, Google search is… just lol! Well, try it till you feel the amazement. Here are some sample pictures of it.


Wolfram Alpha does not just add the directories of the websites . but it relies on licensed data catalogues by its employees, that are as accurate it can ever be. Unlike a normal search engine, it does not give you pictures of celebrities but rather gives you factual answer as well as its visual interpretation. (Phew!! Google is out of danger!..hmmm..maybe).

  • Blue Brain Project : It is a project aiming to reverse engineer the brain. It reconstructs the brain ‘piece by piece’. The cortex in the brain is the most evolved and advanced technology created by nature, and this project aims to virtually simulate that. They initially started with a rat’s brain and have successfully using 100 mesocircuits, and by 2023, they have planned to recreate the human brain which is equivalent too 1000 rat brains.To further the understanding they have planned to simulate at a molecular level to study the chemistry behind cognition.
  • A.L.I.C.E : It is an acronym for “Artificial Linguistic logoInternet Computer Entity” is a chatterbot, means you can have a conversation with a computer that uses natural language processing. It is an open source software. The researches created a XML-compliant language called AIML (AI markup language) in which even users can customize their chatterbox.

I don’t think i need to explain about IBM’s two most mind-blowing machines Deep Blue and Watson.

So, after this quite a lengthy introduction to AI, we shall come to the topic, what is entropy? and How is it even related to AI?.. That will be in the next Adhyayam!!

“You should never be surprised by or feel the need to explain why any physical system is in a high entropy state.”
― Brain Greene, The Fabric of the Cosmos

For Further Readings :

Kilobots :        http://spectrum.ieee.org/automaton/robotics/robotics-hardware/a-thousand-kilobots-self-assemble                 http://www.eecs.harvard.edu/ssr/projects/progSA/kilobot.html

Wolfram Alpha : http://computer.howstuffworks.com/internet/basics/wolfram-alpha1.htm                                                         http://www.wolframalpha.com/

Blue Brain Project : http://bluebrain.epfl.ch/page-52063.html