Indeed it works great for babble. OpenAI suggests the prompt of “Write a tagline for an ice cream shop” as an example. Using it I was able to generate the following three examples:
The best ice cream in town!
A scoop of happiness.
Delicious ice cream for all!
I would have assumed that you could do pruning just by prompting it to generate an evaluation and a rating. Evaluating the above taglines using the prompt “Explain the advantages and disadvantages of the following tagline for an ice cream shop, and then rate the tagline on a scale from 1 to 10.”, I get:
The best ice cream in town!:
The advantages of the tagline are that it is short, sweet, and to the point. It tells customers exactly what they can expect from the ice cream shop. The disadvantages are that it is somewhat generic and does not really stand out from other ice cream shops. Overall, I would rate the tagline a 7 out of 10.
The advantage of the tagline is that it is succinct and to the point. The disadvantage is that it is vague and does not give any specific information about the ice cream shop. I would rate the tagline a 6.
The tagline is straightforward and to the point. It tells customers what they can expect from the ice cream shop. However, it is a bit generic and could apply to any ice cream shop. Rating: 7
A scoop of happiness.:
The advantage of the tagline is that it is short, simple, and to the point. It tells customers exactly what they can expect from the ice cream shop – happiness. The disadvantage of the tagline is that it is a bit cliché and might not stand out from the competition. Overall, I would rate the tagline a 7.
The advantage of the tagline is that it is short and catchy. The disadvantage is that it is somewhat corny. I would rate the tagline a 6.
The tagline is short, catchy, and easy to remember. It conveys a positive message about the product. However, it is somewhat generic and could be used for any type of business. 7⁄10
Delicious ice cream for all!:
The tagline is simple and straightforward, and it tells customers what they can expect from the shop. However, it is somewhat generic and does not really stand out from the crowd. I would rate it a 6.
The tagline is simple and straightforward, which is good. However, it is also somewhat generic and does not really convey anything unique about the shop. It is not a bad tagline, but it is not particularly great either. I would give it a 6 out of 10.
The tagline is straightforward and to the point. It tells customers that the shop sells delicious ice cream and that everyone is welcome. However, it is somewhat generic and doesn’t really stand out from other ice cream shops. It also doesn’t say anything about the quality of the ingredients or the variety of flavors available. Overall, I would rate this tagline a 6 out of 10.
😅I suppose this doesn’t tell us much. The evaluations are all very repetitive, but they seem to correctly apply to the generated ideas, it’s just that the original ideas are not very distinct.
To try and generate more distinct ideas, I asked GPT-3 to come up with some themes for ice cream shops. It gave the following themes:
The ice cream shop would have a tropical theme! You could have all sorts of fruity and exotic flavors, and the decor could be bright and colorful.
The ice cream shop could be called “Frosty’s” and would have a winter theme. The walls would be painted blue and white, and the floors would be covered in fake snow. The ceiling would be covered in twinkling lights, and the ice cream would be served in bowls made to look like snowballs.
The shop would be decorated like an old-fashioned ice cream parlor, with checkered tablecloths, vintage ice cream posters, and a staff dressed in traditional ice cream server uniforms. The menu would feature classic ice cream flavors like vanilla, chocolate, and strawberry, as well as more unique flavors like lavender, Earl Grey, and fig.
This yielded:
Tropical: The advantages of a tropical-themed ice cream shop would be the wide variety of flavors to choose from, as well as the bright and colorful decor. The disadvantages could include the potential for the shop to be too noisy or crowded, and the lack of variety in terms of toppings and mix-ins. Overall, I would rate this theme a 7.
The advantages of the theme are that it would be unique and eye-catching. It would also be appropriate for the winter season. The disadvantages are that it could be difficult to keep the floors clean, and the ceiling lights could be a fire hazard. I would rate the theme an 8 out of 10.
The advantages of this theme are that it would be unique and would likely attract customers who are looking for a more traditional ice cream experience. The disadvantages are that it would be more expensive to set up and maintain, and the menu would be limited to classic and unique flavors, which may not be what some customers are looking for. Overall, I would rate this theme a 7.
This critique doesn’t seem great. For instance, ice cream is disproportionately eaten in the summer, yet it praises “Frosty’s” for being “appropriate for the winter season”. Possibly my prompt is bad though.
But at a first look, it seems like you might be right? Idk.
Explain the advantages for each of the following taglines for an ice cream shop, and evaluate which one is the best: a. The best ice cream in town! b. A scoop of happiness. c. Delicious ice cream for all!
I’m getting the following results:
Attempt one:
The first tagline, “The best ice cream in town!” is a strong claim that could help to attract customers. However, it could also be seen as arrogant and could turn some customers away. The second tagline, “A scoop of happiness,” is more positive and upbeat, and could help to create a positive association with the ice cream shop. The third tagline, “Delicious ice cream for all!,” is inclusive and could help to attract a wide range of customers. Overall, the second tagline, “A scoop of happiness,” is the best option.
Attempt two:
The best ice cream in town! - This tagline is straightforward and to the point. It tells customers that the shop has the best ice cream in town, and that they can expect a delicious treat.
A scoop of happiness. - This tagline is more whimsical and fun. It suggests that the ice cream shop is a place where customers can find happiness, and that their ice cream will be a delicious part of that experience.
Delicious ice cream for all! - This tagline is inclusive and welcoming. It tells customers that everyone is welcome at the shop, and that they can expect to find delicious ice cream.
The best one is The best ice cream in town!
Attempt three, changing it from explaining an advantage to explaining a disadvantage:
The disadvantage of the first tagline is that it is generic and does not stand out from other ice cream shops. The disadvantage of the second tagline is that it is too vague and does not tell the customer what they can expect from the shop. The disadvantage of the third tagline is that it is too general and does not focus on a specific type of ice cream. The best tagline would be something that is specific and tells the customer what they can expect from the shop, such as “The best ice cream in town!”
Yeah… I mean it’s not thinking / comparing / reckoning / discerning, it’s just.… saying things that are the sort of thing that someone says in that context...
I guess this would be the expected failure mode, too. I do wonder if people who have had more experience working with LLMs find this to be the failure mode, or if it’s just because I’m not prompting them right.
Yeah, not so impressive or useful-seeming. I would guess someone very skilled at prompting LLMs could get something slightly useful in this genre with a fair amount of work, but not very useful.
An underlying issue is that, as you pointed out elsewhere IIRC, what we’re wanting is the AI’s own dynamic of acting agentically which induces an evaluation of which things are instrumentally useful. That discernment of what’s useful for acting in the world isn’t in GPT, so you can’t evoke it through prompts. So it can’t do the sort of pruning you could do if you have a familiarity with what sort of things are useful in the world. Maybe.
(Also, “A scoop of happiness” is clearly the best one!)
You could maybe use an LM for Babble. But how would you use an LM for Prune?
Indeed it works great for babble. OpenAI suggests the prompt of “Write a tagline for an ice cream shop” as an example. Using it I was able to generate the following three examples:
The best ice cream in town!
A scoop of happiness.
Delicious ice cream for all!
I would have assumed that you could do pruning just by prompting it to generate an evaluation and a rating. Evaluating the above taglines using the prompt “Explain the advantages and disadvantages of the following tagline for an ice cream shop, and then rate the tagline on a scale from 1 to 10.”, I get:
The best ice cream in town!:
The advantages of the tagline are that it is short, sweet, and to the point. It tells customers exactly what they can expect from the ice cream shop. The disadvantages are that it is somewhat generic and does not really stand out from other ice cream shops. Overall, I would rate the tagline a 7 out of 10.
The advantage of the tagline is that it is succinct and to the point. The disadvantage is that it is vague and does not give any specific information about the ice cream shop. I would rate the tagline a 6.
The tagline is straightforward and to the point. It tells customers what they can expect from the ice cream shop. However, it is a bit generic and could apply to any ice cream shop. Rating: 7
A scoop of happiness.:
The advantage of the tagline is that it is short, simple, and to the point. It tells customers exactly what they can expect from the ice cream shop – happiness. The disadvantage of the tagline is that it is a bit cliché and might not stand out from the competition. Overall, I would rate the tagline a 7.
The advantage of the tagline is that it is short and catchy. The disadvantage is that it is somewhat corny. I would rate the tagline a 6.
The tagline is short, catchy, and easy to remember. It conveys a positive message about the product. However, it is somewhat generic and could be used for any type of business. 7⁄10
Delicious ice cream for all!:
The tagline is simple and straightforward, and it tells customers what they can expect from the shop. However, it is somewhat generic and does not really stand out from the crowd. I would rate it a 6.
The tagline is simple and straightforward, which is good. However, it is also somewhat generic and does not really convey anything unique about the shop. It is not a bad tagline, but it is not particularly great either. I would give it a 6 out of 10.
The tagline is straightforward and to the point. It tells customers that the shop sells delicious ice cream and that everyone is welcome. However, it is somewhat generic and doesn’t really stand out from other ice cream shops. It also doesn’t say anything about the quality of the ingredients or the variety of flavors available. Overall, I would rate this tagline a 6 out of 10.
😅I suppose this doesn’t tell us much. The evaluations are all very repetitive, but they seem to correctly apply to the generated ideas, it’s just that the original ideas are not very distinct.
To try and generate more distinct ideas, I asked GPT-3 to come up with some themes for ice cream shops. It gave the following themes:
The ice cream shop would have a tropical theme! You could have all sorts of fruity and exotic flavors, and the decor could be bright and colorful.
The ice cream shop could be called “Frosty’s” and would have a winter theme. The walls would be painted blue and white, and the floors would be covered in fake snow. The ceiling would be covered in twinkling lights, and the ice cream would be served in bowls made to look like snowballs.
The shop would be decorated like an old-fashioned ice cream parlor, with checkered tablecloths, vintage ice cream posters, and a staff dressed in traditional ice cream server uniforms. The menu would feature classic ice cream flavors like vanilla, chocolate, and strawberry, as well as more unique flavors like lavender, Earl Grey, and fig.
This yielded:
Tropical: The advantages of a tropical-themed ice cream shop would be the wide variety of flavors to choose from, as well as the bright and colorful decor. The disadvantages could include the potential for the shop to be too noisy or crowded, and the lack of variety in terms of toppings and mix-ins. Overall, I would rate this theme a 7.
The advantages of the theme are that it would be unique and eye-catching. It would also be appropriate for the winter season. The disadvantages are that it could be difficult to keep the floors clean, and the ceiling lights could be a fire hazard. I would rate the theme an 8 out of 10.
The advantages of this theme are that it would be unique and would likely attract customers who are looking for a more traditional ice cream experience. The disadvantages are that it would be more expensive to set up and maintain, and the menu would be limited to classic and unique flavors, which may not be what some customers are looking for. Overall, I would rate this theme a 7.
This critique doesn’t seem great. For instance, ice cream is disproportionately eaten in the summer, yet it praises “Frosty’s” for being “appropriate for the winter season”. Possibly my prompt is bad though.
But at a first look, it seems like you might be right? Idk.
Another go at it. Using the prompt
I’m getting the following results:
Attempt one:
Attempt two:
Attempt three, changing it from explaining an advantage to explaining a disadvantage:
Yeah… I mean it’s not thinking / comparing / reckoning / discerning, it’s just.… saying things that are the sort of thing that someone says in that context...
I guess this would be the expected failure mode, too. I do wonder if people who have had more experience working with LLMs find this to be the failure mode, or if it’s just because I’m not prompting them right.
Yeah, not so impressive or useful-seeming. I would guess someone very skilled at prompting LLMs could get something slightly useful in this genre with a fair amount of work, but not very useful.
An underlying issue is that, as you pointed out elsewhere IIRC, what we’re wanting is the AI’s own dynamic of acting agentically which induces an evaluation of which things are instrumentally useful. That discernment of what’s useful for acting in the world isn’t in GPT, so you can’t evoke it through prompts. So it can’t do the sort of pruning you could do if you have a familiarity with what sort of things are useful in the world. Maybe.
(Also, “A scoop of happiness” is clearly the best one!)