Undergraduate math major at Carnegie Mellon University, interested in theoretical aspects of AI Safety via Debate, control, game theory, and probabilistic logic. I’m currently doing theoretical AI Safety via Debate research under Professor Vincent Conitzer.
lunatic_at_large
Karma: 152
Awesome post!
I have absolutely no experience with prediction markets, but I’m instinctively nervous about incentives here. Maybe the real-world incentives of market participants could be greater than the play-money incentives? For example, if you’re trying to select people to represent your country at an international competition and the potential competitors have invested their lives into being on that international team and those potential competitors can sign up as market participants (maybe anonymously), then I could very easily imagine those people sabotaging their competitors’ markets and boosting their own with no regard for their post-selection in-market prediction winnings.
For personal stuff (friendship / dating), I have some additional concerns. Suppose person A is open for dating and person B really wants to date them. By punting the decision-making process to the public, person A restricts themselves to working with publicly available information about person B and simultaneously person B is put under pressure to publicly reveal all relevant information. I can imagine a lot of Goodharting on the part of person B. Also, if it was revealed that person C bet against A and B dating and person B found out, I can imagine some … uh … lasting negative emotions between B and C. That possibility could also mess with person C’s incentives. In other words, the market participants with the closest knowledge of A and B also have the most to lose by A and B being upset with their bets and thus face the most misaligned incentives. (Note: I also dislike dating apps and lots of people use those so I’m probably biased here.)
Finally, I can imagine circumstances where publicly revealing probabilities of things can cause negative externalities, especially on mental health. For example, colleges often don’t reveal students’ exact percentage scores on classes even if employers would be interested — the amount of stress that would induce on the student body could result in worse learning outcomes overall. In an example you listed, with therapists/patients, I feel like it might not be great to have someone suffering from anxiety watch their percentage chance of getting an appointment go up and down.
But for circumstances with low stakes (so play money incentives beat real-world incentives) and small negative externalities, such as gym partners, I could imagine this kind of system working really well! Super cool!
[Question] Current State of Probabilistic Logic
I like these observations! As for your last point about ranges and bounds, I’m actually moving towards relaxing those in future posts: basically I want to look at the tree case where you have more than one variable feeding into each node and I want to argue that even if the conditional probabilities are all 0′s and 1′s (so we don’t get any hard bounds with arguments like the one I present here) there can still be strong concentration towards one answer.
Wow, this is exactly what I was looking for! Thank you so much!
Underspecified Probabilities: A Thought Experiment
I also suspect that the evaluation mechanism is going to be very important. I can think of philosophical debates whose resolution could change the impact of an “artifact” by many orders of magnitude. If possible I think it could be good to have several different metrics (corresponding to different objective functions) by which to grade these artifacts. That way you can give donors different scores depending on which metrics you want to look at. For example, you might want different scores for x-risk minimization, s-risk minimization, etc. That still leaves the “[optimize for (early, reliable) evidence of impact] != [optimize for impact]” issue open, of course.
I really like this perspective! Great first post!
Conditionals All The Way Down
Okay that paper doesn’t seem like what I was thinking of either but it references this paper which does seem to be on-topic: https://research.rug.nl/en/publications/justification-by-an-infinity-of-conditional-probabilities
Thanks for the response! I took a look at the paper you linked to; I’m pretty sure I’m not talking about combinatorial explosion. Combinatorial explosion seems to be an issue when solving problems that are mathematically well-defined but computationally intractable in practice; in my case it’s not even clear that these objects are mathematically well-defined to begin with.
The paper https://www.researchgate.net/publication/335525907_The_infinite_epistemic_regress_problem_has_no_unique_solution initially looks related to what I’m thinking, but I haven’t looked at it in depth yet.
Hmmm, I’m still thinking about this. I’m kinda unconvinced that you even need an algorithm-heavy approach here. Let’s say that you want to apply logit, add some small amount of noise, apply logistic, then score. Consider the function on R^n defined as (score function) composed with (coordinate-wise logistic function). We care about the expected value of this function with respect to the probability measure induced by our noise. For very small noise, you can approximate this function by its power series expansion. For example, if we’re adding iid Gaussian noise, then look at the second order approximation. Then in the limit as the standard deviation of the noise goes to zero, the expected value of the change is some constant (something something Gaussian integral) times the Laplacian of our function on R^n times the square of the standard deviation. Thus the Laplacian is very related to this precision we care about (it basically determines it for small noise). For most reasonable scoring functions, the Laplacian should have a closed-form solution. I think that gets you out of having to simulate anything. Let me know if I messed anything up! Cheers!
If my interpretation of precision function is correct then I guess my main concern is this: how are we reaching inside the minds of the predictors to see what their distribution on is? Like, imagine we have an urn with black and red marbles in it and we have a prediction market on the probability that a uniformly randomly chosen marble will be red. Let’s say that two people participated in this prediction market: Alice and Bob. Alice estimated there to be a 0.3269230769 (or approximately 17⁄52) chance of the marble being red because she saw the marbles being put in and there were 17 red marbles and 52 marbles total. Bob estimated there to be a 0.3269230769 chance of the marble being red because he felt like it. Bob is clearly providing false precision while Alice is providing entirely justified precision. However, no matter which way the urn draw goes, the input tuple (0.3269230769, 0) or (0.3269230769, 1) will be the same for both participants and thus the precision returned by any precision function will be the same. This feels to me like a fundamental disconnect between what we want to measure and what we are measuring. Am I mistaken in my understanding? Thanks!
Awesome post! I’m very ignorant of the precision-estimation literature so I’m going to be asking dumb questions here.
First of all, I feel like a precision function should take some kind of “acceptable loss” parameter. From what I gather, to specify the precision you need some threshold in your algorithm(s) for how much accuracy loss you’re willing to tolerate.
More fundamentally, though, I’m trying to understand what exactly we want to measure. The list of desired properties of a precision function feel somewhat pulled out of thin air, and I’d feel more comfortable with a philosophical understanding of where these properties come from. So let’s say we have a set of possible states/trajectories of the world, the world provides us with some evidence , and we’re interested in for some event . Maybe reality has some fixed out there, but we’re not privy to that, so we’re forced to use some “hyperprior” (am I using that word right?) on probability measures over . After conditioning on , we get some probability distribution on , which participants in a prediction market will take the expected value of as their answer. The precision is trying to quantify something like the standard deviation of this probability distribution on values of , right?
P.S. This is entirely a skill issue on my part but I’m not sure what symbols you’re using for precision function and perturbation function. Detexify was of no use. Feel free to enlighten me!
[Question] Lesswrong’s opinion on infinite epistemic regress and Bayesianism
I’m probably the least qualified person imaginable to represent “the Lesswrong community” given that I literally made my first post this weekend, but I did get into EA between high school and college and I have some thoughts on the topic.
My gut reaction is that it depends a lot on the kind of person this high schooler is. I was very interested in math and theoretical physics when I started thinking about EA. I don’t think I’m ever going to be satisfied with my life unless I’m doing work that’s math-heavy. I applied to schools with good AI programs with the intent of upskilling on AI/ML during college and then going into AI Safety. When I started college I waved away the honors math classes with the intent of getting into theoretical machine learning research as fast as possible. Before the end of freshman year, I realized that I was miserable and the courses felt dumb and that I was finding it very hard to relate to any of the other people in the AI program—most of them were practically-minded and non-math-y. I begged to be let back into the honors math courses and thankfully the department allowed me to do so. I proceeded to co-found the AI Safety club at my college and have been thinking somewhat independently on questions adjacent to AI Safety that interest me. In retrospect, I think that I was too gung-ho about upskilling on ML to stop and pay attention to where my skills and my passion were. This nearly resulted in me having no friend group in college and not being productive at anything.
So yeah, I don’t know what exactly I would recommend. If I had been a more practically-minded person then my actions would probably have been pretty perfect. I guess the only advice I can give is cliches: think independently, explore, talk to people, listen to yourself. Sorry I can’t say anything more concrete!
You raise an excellent point! In hindsight I’m realizing that I should have chosen a different example, but I’ll stick with it for now. Yes, I agree that “What states of the universe are likely to result from me killing vs not killing lanternflies” and “Which states of the universe do I prefer?” are both questions grounded in the state of the universe where Bayes’ rule applies very well. However, I feel like there’s a third question floating around in the background: “Which states of the universe ‘should’ I prefer?” Based on my inner experiences, I feel that I can change my values at will. I specifically remember a moment after high school when I first formalized an objective function over states of the world, and this was a conscious thing I had to do. It didn’t come by default. You could argue that the question “Which states of the universe would I decide I should prefer after thinking about it for 10 years” is a question that’s grounded in the state of the universe so that Bayes’ Rule makes sense. However, trying to answer this question basically reduces to thinking about my values for 10 years; I don’t know of a way to short circuit that computation. I’m reminded of the problem about how an agent can reason about a world that it’s embedded inside where its thought processes could change the answers it seeks.
If I may propose another example and take this conversation to the meta-level, consider the question “Can Bayes’ Rule alone answer the question ‘Should I kill lanternflies?’?” When I think about this meta-question, I think you need a little more than just Bayes’ Rule to reason. You could start by trying to estimate P(Bayes Rule alone solves the lanternfly question), P(Bayes Rule alone solves the lanternfly question | the lanternfly question can be decomposed into two separate questions), etc. The problem is that I don’t see how to ground these probabilities in the real world. How can you go outside and collect data and arrive at the conclusion “P(Bayes Rule alone solves the lanternfly question | the lanternfly question can be decomposed into two separate questions) = 0.734”?
In fact, that’s basically the issue that my post is trying to address! I love Bayes’ rule! I love it so much that the punchline of my post, the dismissive growth-consistent ideology weighting, is my attempt to throw probability theory at abstract arguments that really didn’t ask for probability theory to be thrown at them. “Growth-consistency” is a fancy word I made up that basically means “you can apply probability theory (including Bayes’ Rule) in the way you expect.” I want to be able to reason with probability theory in places where we don’t get “real probabilities” inherited from the world around us.
Hey, thanks for the response! Yes, I’ve also read about Bayes’ Theorem. However, I’m unconvinced that it is applicable in all the circumstances that I care about. For example, suppose I’m interested in the question “Should I kill lanternflies whenever I can?” That’s not really an objective question about the universe that you could, for example, put on a prediction market. There doesn’t exist a natural function from (states of the universe) to (answers to that question). There’s interpretation involved. Let’s even say that we get some new evidence (my post wasn’t really centered on that context, but still). Suppose I see the news headline “Arkansas Department of Stuff says that you should kill lanternflies whenever you can.” How am I supposed to apply Bayes’ rule in this context? How do I estimate P(I should kill lanternflies whenever I can | Arkansas Department of Stuff says I should kill lanternflies whenever I can)? It would be nice to be able to dismiss these kinds of questions as ill-posed, but in practice I spend a sizeable fraction of my time thinking about them. Am I incorrect here? Is Bayes’ theorem more powerful than I’m realizing?
(1) Yeah, I’m intentionally inserting a requirement that’s trivially true. Some claims will make object-level statements that don’t directly impose restrictions on other claims. Since these object-level claims aren’t directly responsible for putting restrictions on the structure of the argument, they induce trivial clauses in the formula.
(2) Absolutely, you can’t provide concrete predictions on how beliefs will evolve over time. But I think you can still reason statistically. For example, I think it’s valid to ask “You put ten philosophers in a room and ask them whether God exists. At the start, you present them with five questions related to the existence of God and ask them to assign probabilities to combinations of answers to these questions. After seven years, you let the philosophers out and again ask them to assign probabilities to combinations of answers. What is the expected value of the shift (say, the KL divergence) between the original probabilities and the final probabilities?“ I obviously cannot hope to predict which direction the beliefs will evolve, but the degree to which we expect them to evolve seems more doable. Even if we’ve updated so that our current probabilities equal the expected value of our future probabilities, we can still ask about the variance of our future probabilities. Is that correct or am I misunderstanding something?
Thanks again, by the way!
So I’ve been thinking a little more about the real-world-incentives problem, and I still suspect that there are situations in which rules won’t solve this. Suppose there’s a prediction market question with a real-world outcome tied to the resulting market probability (i.e. a relevant actor says “I will do XYZ if the prediction market says ABC”). Let’s say the prediction market participants’ objective functions are of the form play_money_reward + real_world_outcome_reward. If there are just a couple people for whom real_world_outcome_reward is at least as significant as play_money_reward and if you can reliably identify those people (i.e. if you can reliably identify the people with a meaningful conflict of interest), then you can create rules preventing those people from betting on the prediction market.
However, I think that there are some questions where the number of people with real-world incentives is large and/or it’s difficult to identify those people with rules. For example, suppose a sports team is trying to determine whether to hire a star player and they create a prediction market for whether the star athlete will achieve X performance if hired. There could be millions of fans of that athlete all over the world who would be willing to waste a little prediction market money to see that player get hired. It’s difficult to predict who those people are without massive privacy violations—in particular, they have no publicly verifiable connection to the entities named in the prediction market.