Request Dall-E image creation via sgpt

[bentito]

I think this type of thing should be possible:

tmp  $ sgpt "Ask Dall-E to draw 'a black-clad futuristic space villain that somewhat resembles Darth Vader', download the PNG and save as ~/Downloads/darth.png"
Sorry, but as a text-based AI, I don't have the capability to interact with Dall-E or perform any visual tasks. I can only provide text-based assistance for programming and system administration related queries.

ChatGPT can handle this request, but I'm not sure about the API.

[moritztim]

I believe ChatGPT is instructed to look for Dall-E requests and generate prompts to send to the Dalll-E api. None of this is implemented in the API afaik. So this would all have to be implemented client side and I don't think that's in the scope of this project.

[bentito]

idk, maybe the tool could catch the "text-based AI..." or mention of Dall-E in the reply and retry to Dall-E API?

This is a fun and handy use case:
if you imagine sgpt in the command pipe before the convert command:

$ convert ~/Downloads/DALL·E\ 2023-12-03\ 19.34.47\ -\ An\ illustration\ of\ a\ futuristic\ space\ villain,\ wearing\ black\ armor\ similar\ in\ style\ to\ traditional\ sci-fi\ characters.\ The\ character\ has\ a\ menacing\ hel.png jpg:- | jp2a --width=80 -

                                '';cc:,'.            .
                             ,cldlOkOkoddc;.        ..
                           .oo,';;:;c;,;;.cdc                 .
                      .   .ol. .,,. . .;;  ,ll        .
                          ll.  . .      .   :o.
                     '   .d:  .  .  .     ...ll
                        .ol... .... . .... ..;o,.
      :               ..lo;'. ..c;'..';c:.. ,.:d'....
        .      .;:c....lo;,;   ..,,...,'..  .,'ld,....c:,.
              .oO.'c',ooc;:   .. .'  ... ..  .c'lx:,,o :lo..
            ..;lk' ;odOc ,.   ... ..' ....    ','lKxx. lod...
          ...'dl';..dxc,...     . .., ..      , .:cdc ..;o:,...
       . ..'coxl. '.:.,   .       ..'..      ..  .,.;.. ;ldd:'..
       ...;oxc:;,'... ..   .   ..........   .    . ...'.';;odo,..
      ..:ldlx,..'.  ... .   .  ..........  .      ..   .'..:dlxl;.
    . ,odxc,..   .....',.::'    ........    .;:,',. ....  ...;ldxl,.
     :l,,'   .  .      ..';.........'...'....';.       .....   '',o;
    ;:,  ..    ....     ..;,  .'';;d0:,,,.  .,,.      ...    .'.  ;cc
   .x.   .'.....          .. ....  .c. ....  ..         ......'.   'x.
 .:do;''..        ..     ..       .';'.       ..    ...        ..',kOd;
::,.   .......  ...       ..... .. ... .  .....       ...  ......,   ';:;
c.    :,...  ..... .    ..   .... ..... .'..  ...   .......   ...:o    ':
  '';;o'. ....    ..   ... .'   ...   ...  ... ...   '...  .... ..l::'.
    .c;''..  ... .       .   . .. ..... .. .  .       . ...   ..'.;c.
    ooc..  ..,,,dk:.     .. ..  .  ...  .;:.   .     .'kk:,,... .'coo
   .do. .........;ol:,.. ... .  ..,:'..';. .. ..  .,:lxl,'. ..... .co;
   l, .   .'.'..  ..,'.';:;.. .',cc;.',......  .,:,,:,..  .'..    ..;l
   l  ...    . ..,,.......;::.,;c;'..,. .   .'c:'...  .',....   ...  c
   l..   .... ..   ...'....' .''.''.,. .   .,,'.....'..   .. .'.     l
   ,:      ...','....  .......c,.... .... .. .......   ...;...     .;c
   o.            .........lc.:dXX:.'.''.   . ...'''......           .l
  ;, .    .                .:KNOc:,,';''.    ,c.    .             .. :.
 .c .'    .   ..         .;0NO;  ;ccc','....             .    .    '  l
 :  '    .    .      . .;ON0:.       ,';;.                .    .    ' .,
.  .    .    .       .,ONKc.  .                    .      .     .    . ..
  .         .       ,kNXl.    '     .     .  .     .       .    .     .
                  'xNXo.     .'           ,
                .dNXd'       .'           ,
               ,0M0:         .'           ,

[moritztim]

While the idea of catching mentions of "text-based AI" or Dall-E in the reply and retrying with Dall-E API sounds cool, it might get messy. Imagine asking "What's Dall-E" and getting a picture. Especially since the project has GPT in its name (and not ChatGPT), implying it prompts GPT, I don't think it's a good idea to have a bodge like this in there.
Instead, there could be a role to generate dall-e prompts. I don't know if that should be included in the repo tho, since it's a very specific use case. You could make it for youself tho and then just pipe the output to some Dall-E client.

Another issue with implementing this is that it's only part of the "workflow" that you have with ChatGPT, where ChatGPT will also see the previous images and then use them to write a better prompt, which would be even more pain to implement.

Rather than implementing a bodge that only has half of the capabilities and might misfire, it's better to just have a feature that allows you to create Dall-E prompts. However since there's currently only 4 very broad roles, I think this would be out of place. But I think this project could benefit from a larger selection of roles, so maybe you could make a feature request for additional roles, to allow everyone to make suggestions!

[moritztim]

#390