(Comment rewritten from scratch after comment editor glitched.)
This article is not about what I expected from the title. I’ve been thinking about “retroactively allocating responsibility”, which sounds a lot like “assigning credit”, in the context of multi-winner voting methods: which voters get credit for ensuring a given candidate won? The problem here is that in most cases no individual voter could change their vote to have any impact whatsoever on the outcome; in ML terms, this is a form of “vanishing gradient”. The solution I’ve arrived at was suggested to me here; essentially, it’s imposing a (random) artificial order on the inputs, then using a model to reason about the “worst”-possible outcome after any subset of the inputs, and giving each of the inputs “credit” for the change in that “worst” outcome.
I’m not sure if this is relevant to the article as it stands, but it’s certainly relevant to the title as it stands.
Yeah, this post is kind of a mix between (first) actually trying to explain/discuss the credit assignment problem, and (second) some far more hand-wavy thoughts relating to myopia / partial agency. I have a feeling that the post was curated based on the first part, whereas my primary goal in writing was actually the second part.
In terms of what I talked about (iirc—not re-reading the post for the sake of this comment), it seems like you’re approaching a problem of how to actually assign credit once you have a model. This is of course an essential part of the problem. I would ask questions like:
Does the proposed formula allow us to construct a mechanism which incentivizes a desirable type of behavior?
Does it encourage honesty?
Does it encourage strategically correcting things toward the best collective outcome (even if dishonestly)?
These are difficult problems. However, my post was more focused on the idea that you need a model at all in order to do credit assignment, which I feel is not obvious to many people.
(Note that we recently pushed a change that allows you to delete your own comments if they have no children, so if you’d like you can delete your earlier comment.)
(Comment rewritten from scratch after comment editor glitched.)
This article is not about what I expected from the title. I’ve been thinking about “retroactively allocating responsibility”, which sounds a lot like “assigning credit”, in the context of multi-winner voting methods: which voters get credit for ensuring a given candidate won? The problem here is that in most cases no individual voter could change their vote to have any impact whatsoever on the outcome; in ML terms, this is a form of “vanishing gradient”. The solution I’ve arrived at was suggested to me here; essentially, it’s imposing a (random) artificial order on the inputs, then using a model to reason about the “worst”-possible outcome after any subset of the inputs, and giving each of the inputs “credit” for the change in that “worst” outcome.
I’m not sure if this is relevant to the article as it stands, but it’s certainly relevant to the title as it stands.
Yeah, this post is kind of a mix between (first) actually trying to explain/discuss the credit assignment problem, and (second) some far more hand-wavy thoughts relating to myopia / partial agency. I have a feeling that the post was curated based on the first part, whereas my primary goal in writing was actually the second part.
In terms of what I talked about (iirc—not re-reading the post for the sake of this comment), it seems like you’re approaching a problem of how to actually assign credit once you have a model. This is of course an essential part of the problem. I would ask questions like:
Does the proposed formula allow us to construct a mechanism which incentivizes a desirable type of behavior?
Does it encourage honesty?
Does it encourage strategically correcting things toward the best collective outcome (even if dishonestly)?
These are difficult problems. However, my post was more focused on the idea that you need a model at all in order to do credit assignment, which I feel is not obvious to many people.
(Note that we recently pushed a change that allows you to delete your own comments if they have no children, so if you’d like you can delete your earlier comment.)