I got frustrated reading this post after only a paragraph or two, and went to downvote it. However, I’ve recently seen comments that people on this site are too quick to downvote. So instead, I’m leaving this comment and not voting. Hopefully that’s a little better than total lack of feedback; you can validly ignore claude’s response and only consider my prompt for what I’m looking for.
Me to Claude Opus:
This is a lesswrong post. Write a harshly honest comment about the post’s flaws; be kind to the author, but precise in your criticism. Is there mathematical grounding for these claims? Does the philosophy have mechanistic basis? Write densely and tersely, many people will read your reply. If I quote you, I’ll cite it being from you.
Claude Opus:
While the author makes an interesting attempt to apply semiotics to critique the orthogonality thesis, the argument ultimately lacks rigor and convincing mathematical or mechanistic grounding.
The central claim that intelligence and final goals must necessarily be correlated in terms of complexity is asserted but not robustly proven. The author posits that more intelligent agents will have more complex symbolic representations of goals, but fails to rule out the possibility of a highly capable agent pursuing a simply represented objective. Quantifying the “complexity” of a goal representation seems crucial to the argument but is not adequately formalized.
Additionally, the idea that incoherent and inconsistent utility functions are required for open-ended curiosity and that signification itself must be rewarded to avoid trading off exploration and achievement is intriguing but underdeveloped. These suggestions require much more detailed proposals for value learning and agent architecture to be actionable alternatives to the coherent utility functions and exploration/exploitation tradeoffs common in current AI systems.
While thought-provoking, the post falls short of dismantling Bostrom’s arguments and would need to engage the technical AI safety literature more deeply to move beyond speculation. The humanistic semiotic perspective is a valuable complement to mathematical and empirical analysis of AI motivations, but cannot replace the need for formal theories and concrete interpretability research to ground concerns about misaligned superintelligent agents.
I applaud the author’s effort to broaden the discourse around AI goals and values, but believe significant elaboration is required for this line of critique to seriously impact the assumptions and priorities of the AI safety community. Maintaining philosophical humility while striving for greater rigor could strengthen future work in this vein.
Also, semiotics was deeply connected to information theory and cybernetics, the fundamental characteristics of signs I bring up here as being distinct come from the cybernetic construct of variety.
You can reason about a topic without “the math”, if anyone can poke actual holes in my logic I would love to hear a substantial objection. It also wasn’t my intention to write nothing but an introduction to semiotics in terms of math.
seems people didn’t like me posting claude, either. I figured they wouldn’t, it’s kind of rude of me to not read it and just dump some ai feedback. seems less rude than a downvote, though.
I got frustrated reading this post after only a paragraph or two, and went to downvote it. However, I’ve recently seen comments that people on this site are too quick to downvote. So instead, I’m leaving this comment and not voting. Hopefully that’s a little better than total lack of feedback; you can validly ignore claude’s response and only consider my prompt for what I’m looking for.
Me to Claude Opus:
Claude Opus:
Seems that Claude, besides not really understanding the arguments I’m making, argues from assertion much more than I do.
Also, semiotics was deeply connected to information theory and cybernetics, the fundamental characteristics of signs I bring up here as being distinct come from the cybernetic construct of variety.
to be useful to me, I’d want to see a shorter, denser post with that introduces the math of the relevant topics.
You can reason about a topic without “the math”, if anyone can poke actual holes in my logic I would love to hear a substantial objection. It also wasn’t my intention to write nothing but an introduction to semiotics in terms of math.
seems people didn’t like me posting claude, either. I figured they wouldn’t, it’s kind of rude of me to not read it and just dump some ai feedback. seems less rude than a downvote, though.