sen

Karma: 171

sen Dec 3, 2016, 8:36 AM
2 points
in reply to: ArisKatsaris’s comment on: December 2016 Media Thread
Some short papers worth reading.

A good introduction to topos theory, which in turn explains why the Yoneda embedding so so useful. I would not recommend this as an introduction to category theory or the Yoneda lemma.

A love letter to adjoint functors discussing their meaning and philosophical significance.

A paper on categorification, from which I will include a quote below in hopes of showing some of you heathens the light.

If one studies categorification one soon discovers an amazing fact: many deep-sounding results in mathematics are just categorifications of facts we learned in high school! There is a good reason for this. All along, we have been unwittingly ‘decategorifying’ mathematics by pretending that categories are just sets. We ‘decategorify’ a category by forgetting about the morphisms and pretending that isomorphic objects are equal. We are left with a mere set: the set of isomorphism classes of objects.

To understand this, the following parable may be useful. Long ago, when shepherds wanted to see if two herds of sheep were isomorphic, they would look for an explicit isomorphism. In other words, they would line up both herds and try to match each sheep in one herd with a sheep in the other. But one day, along came a shepherd who invented categorification. She realized one could take each herd and ‘count’ it, setting up an isomorphism between it and some set of ‘numbers’, which were nonsense words like ‘one, two, three, . . . ’ specially designed for this purpose. By comparing the resulting numbers, she could show that two herds were isomorphic without explicitly establishing an isomorphism! In short, by decategorifying the category of finite sets, the set of natural numbers was invented.

According to this parable, decategorification started out as a stroke of mathematical genius. Only later did it become a matter of dumb habit, which we are now struggling to overcome by means of categorification. While the historical reality is far more complicated, categorification really has led to tremendous progress in mathematics during the 20th century. For example, Noether revolutionized algebraic topology by emphasizing the importance of homology groups. Previous work had focused on Betti numbers, which are just the dimensions of the rational homology groups. As with taking the cardinality of a set, taking the dimension of a vector space is a process of decategorification, since two vector spaces are isomorphic if and only if they have the same dimension. Noether noted that if we work with homology groups rather than Betti numbers, we can solve more problems, because we obtain invariants not only of spaces, but also of maps. In modern parlance, the nth rational homology is a functor defined on the category of topological spaces, while the nth Betti number is a mere function defined on the set of isomorphism classes of topological spaces. Of course, this way of stating Noether’s insight is anachronistic, since it came before category theory. Indeed, it was in Eilenberg and Mac Lane’s subsequent work on homology that category theory was born!

sen Dec 3, 2016, 7:50 AM
13 points
on: Which areas of rationality are underexplored? - Discussion Thread
Non-bayesian reasoning. Seriously, pretty much everything here is about experimentation, conditional probabilities, and logical fallacies, and all of the above are derived from bayesian reasoning. Yes, these things are important, but there’s more to science and modeling than learning to deal with uncertainty.

Take a look at the Wikipedia page on the Standard Model of particle physics, and count the number of times uncertainty and bayesian reasoning are mentioned. If your number is greater than zero, then they must have changed the page recently. Bayesian reasoning tells you what to expect given an existing set of beliefs. It doesn’t tell you how to develop those underlying beliefs in the first place. For much of physics, that’s pretty much squarely in the domain of group theory / symmetry. It’s ironic that a group so heavily based on the sciences doesn’t mention this at all.

Rationality is about more than empirical studies. It’s about developing sensible models of the world. It’s about conveying sensible models to people in ways that they’ll understand them. It’s about convincing people that your model is better than theirs, sometimes without having to do an experiment.

It’s not like these things aren’t well-studied. It’s called math, and it’s been studied for thousands of years. Everything on this site focuses on one tiny branch, and there’s so much more out there.

Apologies for the rant. This has been bugging me for a while now. I tried to create a thread on this a little while ago and met with the karma limitation. I didn’t want to deal with it at the time, and now it’s all coming back to me, rage and all.

Also, this discussion topic is suboptimal if your aim is to explore new areas of rationality, as it presumes that all unexplored areas will arise from direct discussion. It should have been paired with the question “How do we discover underexplored areas of rationality?” My answer is to that is to encourage non-rational discussion where people believe, intuitively or otherwise, that it should be possible to make the discussion rational. You’re not going to discover the boundaries of rationality by always staying within them. You need to look both outside and inside to see where the boundary might lie, and you need to understand non-rationality if you ever want hope of expanding the boundaries of rationality.

End rant.

sen Nov 29, 2016, 8:58 AM
0 points
on: Mismatched Vocabularies
I don’t think there’s anything wrong with comparing 1 and 3. Yes, Reaction 1 is defined by an ideal, Reaction 2 is defined by a goal, and Reaction 3 is defined by an evolutionary impulse (whatever that means), but that does not make these things incomparable. If you have a goal in mind, you can determine the relationship between these three Reactions to the goal, and you can hope to come up with some ordering of Reactions with respect to that goal. For example, if you want to judge these reactions in terms of how well they indicate an individual’s strive towards a goal that the reactor would consider worthwhile, I’d argue that Reaction 3 is, with high likelihood, a stupid option.

Regarding your sudden doubt in your own perspective, the problem here is that you didn’t define a goal. By not defining a goal, your implicit goal was allowed to become something you didn’t understand, and the reasoning behind your judgement was allowed to become highly subjective and non-conveyable. You can fix this particular issue by making sure to always think of “goodness” and “badness” in relation to explicitly specified goals. The more general issue is that you don’t have a basis for recognizing when you believe things are reasonably comparable. You can fix this more general issue by studying more math, specifically category theory.

Regarding your actual question, try redefining your three reactions in terms of each of the three properties you used to define them: reaction from ideal, reaction from goal, reaction from evolutionary impulse (whatever that means). Under what ideal is it correct to ignore the mismatch and carry on the conversation? (I think korin43 answered most of this question.) Under what ideal is it correct to display anger? Towards what ends is it good to Google the unknown reference before responding? Towards what ends is it good to get angry?

As a side note, I’ve never seen evolutionary anything used as a concrete justification for a phenomenon where it couldn’t equally well be used to justify the lack of the same phenomenon. More often than not, I see it as an attempt to hand-wave away a complex behavior because thinking is hard.

sen Feb 28, 2016, 11:48 AM
0 points
in reply to: Luke_A_Somers’s comment on: Goal completion: noise, errors, bias, prejudice, preference and complexity
I see. You’re treating “energy” as the information required to specify a model. Your analogy and your earlier posts make sense now.

sen Feb 25, 2016, 6:21 AM
0 points
in reply to: Luke_A_Somers’s comment on: Goal completion: noise, errors, bias, prejudice, preference and complexity
2) I think this is the distinction you are trying to make between the lattice model and the smoker model: in the lattice model, the equations and parameters are defined, whereas in the smoker model, the equations and parameters have to be deduced. Is that right? If so, my previous posts were referring to the smoker-type model.

Your toy meta-model is consistent with what I was thinking when I used the word “model” in my previous comments.

3) I see what you’re saying. If you add complexity to the model, you want to make sure that its improvement in ability is greater than the amount of complexity added. You want to make sure that the model isn’t just “memorizing” the correct results, and that all model complexity comes with some benefit of generalizability.

I don’t think temperature is the right analogy. What you want is to penalize a model that is too generally applicable. Here is a simple case:

simple case A one-hidden-layer feed-forward binary stochastic neural network the goal of which is to find binary-vector representations of its binary-vector inputs. It translates its input to an internal representation of length n, then translates that internal representation into some binary-vector output that is the same length as its input. The error function is the reconstruction error, measured as the KL-divergence from input to output.

The “complexity” you want is the length of its internal representation in unit bits since each element of the internal representation can retain at most one bit of information, and that bit can be arbitrarily reflected by the input. The information loss is the same as the reconstruction error in unit bits since that describes the probability of the model guessing correctly on a given input stream (assuming each bit is independent). Your criterion translates to “minimize reconstruction error + internal representation size”, and this can be done by repeatedly increasing the size of the internal representation until adding one more element reduces reconstruction error by less than one bit.

sen Feb 24, 2016, 2:05 AM
0 points
in reply to: Luke_A_Somers’s comment on: Goal completion: noise, errors, bias, prejudice, preference and complexity
I thought you were referring to degenerate gases when you mentioned nontrivial behavior in solid state systems since that is the most obvious case where you get behavior that cannot be easily explained by the “obvious” model (the canonical ensemble). If you were thinking of something else, I’m curious to know what it was.

I’m having a hard time parsing your suggestion. The “dropout” method introduces entropy to “the model itself” (the conditional probabilities in the model), but it seems that’s not what you’re suggesting. You can also introduce entropy to the inputs, which is another common thing to do during training to make the model more robust. There’s no way to introduce 1 bit of entropy per “1 bit of information” contained in the input though since there’s no way to measure the amount of information contained in the input without already having a model of the input. I think systematically injecting noise into the input based on a given model is not functionally different from injecting noise into the model itself, at least not in the ideal case where the noise is injected evenly.

You said that “if you add 1 bit of information, you have added 1 bit of entropy”. I can’t tell if you’re equating the two phrases or if you’re suggesting adding 1 bit of entropy for every 1 bit of information. In either case, I don’t know what it means. Information and entropy are negations of one another, and the two have opposing effects on certainty-of-an-outcome. If you’re equating the two, then I suspect you’re referring to something specific that I’m not seeing. If you’re suggesting adding entropy for a given amount of information, it may help if you explain which probabilities are impacted. To which probabilities would you suggest adding entropy, and which probabilities have information added to them?

sen Feb 23, 2016, 4:23 AM
0 points
in reply to: Luke_A_Somers’s comment on: Goal completion: noise, errors, bias, prejudice, preference and complexity
It’s true that the probability of a microstate is determined by energy and temperature, but the Maxwell-Boltzmann equation assumes that temperature is constant for all particles. Temperature is a distinguishing feature of two distributions, not of two particles within a distribution, and least-temperature is not a state that systems tend towards.

As an aside, the canonical ensemble that the Maxwell-Boltzmann distribution assumes is only applicable when a given state is exceedingly unlikely to be occupied by multiple particles. The strange behavior of condensed matter that I think you’re referring to (Bose-Einstein condensates) is a consequence of this assumption being incorrect for bosons, where a stars-and-bars model is more appropriate.

It is not true that information theory requires the conservation of information. The Ising Model, for example, allows for particle systems with cycles of non-unity gain. This effectively means that it allows particles to act as amplifiers (or dampeners) of information, which is a clear violation of information conservation. This is the basis of critical phenomena, which is a widely accepted area of study within statistical mechanics.

I think you misunderstand how models are fit in practice. It is not standard practice to determine the absolute information content of input, then to relay that information to various explanators. The information content of input is determined relative to explanators. However, there are training methods that attempt to reduce the relative information transferred to explanators, and this practice is called regularization. The penalty-per-relative-bit approach is taken by a method called “dropout”, where a random “cold” model is trained on each training sample, and the final model is a “heated” aggregate of the cold models. “Heating” here just means cutting the amount of information transferred from input to explanator by some fraction.

sen Feb 22, 2016, 7:40 AM
0 points
in reply to: Luke_A_Somers’s comment on: Goal completion: noise, errors, bias, prejudice, preference and complexity
How is inverse temperature a penalty on models? If you’re referring to the inverse temperature in the Maxwell-Boltzmann distribution, the temperature is considered a constant, and it gives the likelihood of a particle having a particular configuration, not the likelihood of a distribution.

Also, I’m not sure it’s clear what you mean by “information to specify [a model]”. Does a high inverse temperature mean a model requires more information, because it’s more sensitive to small changes and therefore derives more information from them, or does it mean that the model requires less information, because it derives less information from inputs?

The entropy of the Maxwell-Boltzmann distribution I think is proportional to log-temperature, so high temperature (low sensitivity to inputs) is preferred if you go strictly by that. People that train neural networks generally do this as well to prevent overtraining, and they call it regularization.

If you are referring to the entropy of a model, you penalize a distribution for requiring more information by selecting the distribution that maximizes entropy subject to whatever invariants your model must abide by. This is typically done through the method of Lagrange multipliers.