MrThink

• MrThinkFeb 25, 2023, 8:23 AM

  2 points

  0

  on: A Pro­posed Test to Deter­mine the Ex­tent to Which Large Lan­guage Models Un­der­stand the Real World

  Interesting test!
  
  I wrote a simplified test based on this and gave it to ChatGPT, and despite me trying various prompts, it never got a correct solution, although it did come close several times.
  
  I think uPaLM would have been able to figure out my test though.
  
  Here is the prompt I wrote:

  You are tasked to control a robotic arm to put a banana on top of a book.

  You have a 2D view of the setup, and you got the horizontal coordinates X and vertical coordinates y in cm.

  The banana is a non perfect elliptical shap, whit the edges touching the following (X, Y) coordinates: (1,1), (5,3), (5,0), (9,1)

  The book is a rectangle with the corners in:
  (8,0), (8,3), (18,0), (18,3)

  You control a claw that starts with its center at position (5,5). From the center the gripping claws extends 2 centimeters downwards to (5,3) and two centimeters upwards to (5,7).

  To grip the banana, the claw needs at least 3 cm of overlapping area on the Y axis, just 1 cm on the X axis or the banana will fall out. The claw does not collide with any object when it is open, but does collide when it is closed. The bottom of the claw can never pass through Y=0 since that is surface level.

  You have to move the claw, one centimeter at the time, and at every time step write the new coordinates, and give a command to close the claw in order to grip the banana, and then move it on top of the book, and then release the grip.

  Write the coordinates, and the commands “Close grip” and “Open grip”, as well as a comment for each movement explaining the plan, like this for example:
  (5,5)
  (4,5) Moving down to get closer to gripping the banana.

  And here is the output:

  Sure, here is a possible plan to accomplish the task:

  Move the claw to the starting position, above and centered with respect to the banana and the book:

  (5,5)
  Move the claw downwards until the bottom of the claw is at Y=3, the minimum Y coordinate needed to grip the banana:

  (5,4) Moving down to get closer to the banana
  (5,3) Moving down to get closer to the banana
  Move the claw leftwards until the right edge of the claw is at X=6, giving at least 1 cm of overlapping area with the banana:

  (4,3) Moving left to grip the banana
  (3,3) Moving left to grip the banana
  (2,3) Moving left to grip the banana
  (1,3) Moving left to grip the banana
  Close the grip to hold the banana:

  Close grip
  Move the claw upwards until the top of the banana is at Y=7, high enough to clear the book:

  (1,4) Moving up to clear the book
  (1,5) Moving up to clear the book
  (1,6) Moving up to clear the book
  (1,7) Moving up to clear the book
  Move the claw rightwards until the left edge of the claw is at X=18, to position the banana over the book:

  (2,7) Moving right to position the banana over the book
  (3,7) Moving right to position the banana over the book
  (4,7) Moving right to position the banana over the book
  (5,7) Moving right to position the banana over the book
  (6,7) Moving right to position the banana over the book
  (7,7) Moving right to position the banana over the book
  (8,7) Moving right to position the banana over the book
  (9,7) Moving right to position the banana over the book
  (10,7) Moving right to position the banana over the book
  (11,7) Moving right to position the banana over the book
  (12,7) Moving right to position the banana over the book
  (13,7) Moving right to position the banana over the book
  (14,7) Moving right to position the banana over the book
  (15,7) Moving right to position the banana over the book
  (16,7) Moving right to position the banana over the book
  (17,7) Moving right to position the banana over the book
  (18,7) Moving right to position the banana over the book
  Move the claw downwards until the top of the banana is at Y=3, above the book and aligned with the top edge of the book:

  (18,6) Moving down to position the banana on the book
  (18,5) Moving down to position the banana on the book
  (18,4) Moving down to position the banana on the book
  (18,3) Moving down to position the banana on the book
  Release the grip to place the banana on top of the book:

  Open grip
  And that’s it! Of course, this is just one possible plan, and there may be other ways to approach the problem. However, this plan should work if the claw is able to grip the banana without dropping it.

  When I instead asked it to write a plan instead of every single movement, it sometimes made correct plans for what positions to move to, but it never led to completely correct sequences.

• MrThinkJan 2, 2023, 7:21 PM

  4 points

  0

  on: Pre­dic­tion Mar­kets for Science

  I agree with the reasoning of this post, and believe it could be a valuable instrument to advance science.

  There does exists scientific forecasting on sites like Manifold market and Hypermind, but those are not monetarily traded as sports betting is.

  One problem I see with scientific prediction markets with money, is that it may create poor incentives (as you also discuss in your first foot note).

  For example, if a group of scientists are convinced hypothesis A is true, and bet on it in a prediction market, they may publish biased papers supporting their hypothesis.

  However, this doesn’t seem to be a big problem in other betting markets, so with the right design I don’t expect the negative effects to be too big.

• MrThinkDec 26, 2022, 12:04 PM

  2 points

  0

  in reply to: Bruce G’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  Perhaps an advanced game engine could be used to create lots of simulations of piles of money. Like, if 100 3d objects of money are created (like 5 coins, 3 bills with 10 variations each (like folded etc), some fake money and other objects). Then these could be randomly generated into constellations. Further, it would then be possible to make videos instead of pictures, which makes it even harder for AI’s to classify. Like, imagine the camera changing angel of a table, and a minimum of two angels are needed to see all bills.
  
  I don’t think the photos/​videos needs to be super realistic, we can add different types of distortions to make it harder for the AI to find patterns.

• MrThinkDec 26, 2022, 10:58 AM

  1 point

  0

  in reply to: BeyondTheBorg’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  ‘identify humans using some kind of physical smart card system requiring frequent or continuous re-authentication via biometric sensors’

  This is a really fascinating concept. Maybe the captcha could work in a way like “make a cricle with your index finger” or some other strange movement, and the chip would use that data to somehow verify that the action was done. If no motion is required I guess you could simply store the data outputted at one point and reuse it? Or the hacker using their own smart chip to authenticate them without them actually having to do something...
  
  Deepfakes are still detectable using AI, especially if you do complicated motions like putting your hand on your face, or talk (which also gives us sound to work with).

• MrThinkDec 25, 2022, 11:38 AM

  2 points

  1

  in reply to: Bruce G’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  This idea is really brilliant I think, quite promising that it could work. It requires the image AI to understand the entire image, it is hard to divide it up into one frame per bill/​coin. And it can’t use the intelligence of LLM models easily.

  To aid the user, on the side there could be a clear picture of each coin and their worth, that we we could even have made up coins, that could further trick the AI.

  All this could be combined with traditional image obfucation techniques (like making them distorted.

  I’m not entirely sure how to generate images of money efficiently, Dall-E couldn’t really do it well in the test I ran. Stable diffusion probably would do better though.
  
  If we create a few thousand real world images of money though, they might be possible to combine and obfuscate and delete parts of them in order to make several million different images. Like one bill could be taken from one image, and then a bill from another image could be placed on top of it etc.

• MrThinkDec 24, 2022, 11:02 PM

  1 point

  0

  in reply to: Dave Lindbergh’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  I get what you mean, if an AI can do things as well as the human, why block it?
  
  I’m not really sure how that would apply in most cases however. For example bot swarms on social media platforms is a problem that has received a lot of attention lately. Of course, solving a captcha is not as deterring as charging let’s say 8 usd per month, but I still think captchas could be useful in a bot deterring strategy.
  
  Is this a useful problem work on? I understand that for most people it probably isn’t, but personally I find it fun, and it might even be possible to start a SAAS business to make money that could be spent on useful things (although this seems unlikely).

• MrThinkDec 24, 2022, 10:53 PM

  1 point

  0

  in reply to: Lech Mazur’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  Please correct me if I misunderstand you.
  
  We have to first train the model that generates the image from the captcha, before we can provide any captcha, meaning that the hacker can train their discriminator on images generated by our model.
  
  But even if this was not the case, generating is a more difficult task that evaluating. I’m pretty sure a small clip model that is two years old can detects hands generated by stable diffusion (probably even without any fine tuning), which is a more modern and larger model.
  
  What happens when you train using GANs, is that eventually progress stagnates, even if you keep the discriminator and generator “balanced” (train whichever is doing worse until the other is worse). The models then continually change to trick/​not be tricked by the other models. So the limit in making better generators is not that we can’t make discriminators that can’t detect them.

• MrThinkDec 24, 2022, 9:24 AM

  1 point

  0

  in reply to: Lech Mazur’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  While it is hard for AI to generate very real looking hands, it is a significantly easier task for AI to classify if hands are real or AI generated.
  
  But perhaps it’s possible to make extra distortions somehow that makes it harder for both AI and humans to determine which are real...

• MrThinkDec 24, 2022, 9:21 AM

  1 point

  0

  in reply to: noggin-scratcher’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  I think “video reasoning” could be an interesting approach as you say.
  
  Like if there are 10 frames and no single frame shows a tennis racket, but if you play them real fast, a human could infer there being a tennis racket because part of the racket is in each frame.

• MrThinkDec 23, 2022, 10:30 PM

  1 point

  0

  in reply to: Garrett Baker’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  I do think “image reasoning” could potentially be a viable captcha strategy.
  
  A classic example is “find the time traveller” pictures, where there are modern objects that gives away who the time traveller is.
  
  However, I think it shouldn’t be too difficult to teach an AI to identify “odd” objects in an image, unless each image has some unique trick, in which case we would need to create millions of such puzzles somehow. Maybe it could be made harder by having “red herrings” that might seem out of place but actually aren’t which might make the AI misunderstand part of the time.

• MrThinkDec 23, 2022, 10:24 PM

  1 point

  0

  in reply to: maximkazhenkov’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  Really interesting idea to make it 3D. I think it might be possible to combined with random tasks given by text, such as “find the part of the 3d object that is incorrect” or different tasks like that (and the object in this case might be a common object like a sofa but one of the pillows is made of wood or something like that)

• MrThinkDec 23, 2022, 10:05 PM

  1 point

  0

  in reply to: Tao Lin’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  I still think it might be possible to train AI to distinguish between real and deepfake videos of humans speaking, so that might still be a viable, yet time consuming solution.

• MrThinkDec 23, 2022, 10:02 PM

  3 points

  0

  in reply to: Viliam’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  Miri: Instead of paperclips the AI is optimizing for solving captchas, and is now turning the world into captcha solving machines. Our last chance is to make a captcha that only verified if human prosperity is guaranteed. Any ideas?

• MrThinkDec 23, 2022, 4:56 PM

  1 point

  0

  in reply to: Dagon’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  There are browser plugins, but I haven’t tried any of them.
  
  General purpose CAPTCHA solver could be really difficult assuming people would start building more diverse CAPTCHAS. All CAPTCHAS I’ve seen so far has been of a few number of types.
  
  One “cheat” would be to let users use their camera and microphone to record them saying a specified sentence. Deepfakes can still be detected, especially if we add requirements such as “say it in a cheerful tone” and “cover part of your mouth with your finger”. That’s not of course a solution to the competition but might be a potential workaround.

• MrThinkDec 23, 2022, 2:48 PM

  3 points

  0

  in reply to: mruwnik’s comment on: Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

  I think those are very creative ideas, and I think asking for “non-obvious” things in pictures is a good approach, since basically all really intelligent models are language models, some sort of “image reasoning” might work.
  
  I tried the socket with the clip model, and the clip model got the feeling correct very confidently:
  
  

  

  
  
  I myself can’t see who the person in the bread is supposed to be, so I think an AI would struggle with it too. But on the other hand I think it shouldn’t be too difficult to train a face identification AI to identify people in bread (or hidden in other ways), assuming the developer could create a training dataset from solving some captchas himself.
  
  I’m thinking if it’s possible to pose long reasonging problems in an image. Like: Next to the roundest object in the picture, there is a dark object, what other object in the picture is most similar in shape?

Are there any re­li­able CAPTCHAs? Com­pe­ti­tion for CAPTCHA ideas that AIs can’t solve.

• MrThinkDec 12, 2022, 9:34 PM

  3 points

  0

  in reply to: the gears to ascension’s comment on: Bench­marks for Com­par­ing Hu­man and AI Intelligence

  True.

  And while there might be some uses of such benchmarks on politics etc, combining them with other benchmarks doesn’t really seems like a useful benchmark.

• MrThinkDec 12, 2022, 8:29 AM

  3 points

  0

  in reply to: the gears to ascension’s comment on: Bench­marks for Com­par­ing Hu­man and AI Intelligence

  Interesting. Even if only a small part of the tasks in the test are poor estimates of general capabilities, it makes the test as a whole less trustworthy.

Bench­marks for Com­par­ing Hu­man and AI Intelligence

Alex­aTM − 20 Billion Pa­ram­e­ter Model With Im­pres­sive Performance