new tool: idle-crafting

[chdoc]

This tool allows idle dwarfs to automatically satisfy their crafting needs.

This isn't ready yet, but given that it involves quite a bit of reverse engineering and guesswork, I think this is the time to get some reviews.

[myk002]

running anything every 53 ticks is likely going to be a performance sink, but from a quick look through this code, it seems reasonable architecture- and logic-wise.

re:performance testing, try this:

1. enable idle-crafting
2. :lua dfhack.internal.resetPerfCounters()
3. run for a few in-game days
4. :lua require('script-manager').print_timers()

[chdoc]

re:performance testing, try this:

Here are some rough ballpark figures.

# idle-crafting status
Fulfillment levels for "craft item" needs
  14 unfettered
   8 level-headed
  10 untroubled
  55 not distracted
  75 unfocused
  26 distracted
Script is enabled with 2 workshops configured for idle crafting
The thresholds for "craft item" needs are 10000/1000/500

Lua timer details
-----------------

                            idle-craftingunit      849 ms ( 83.65% of lua timers,   0.68% of elapsed)
                  control-panel/fix/ownership       56 ms (  5.52% of lua timers,   0.04% of elapsed)
             control-panel/fix/general-strike       19 ms (  1.87% of lua timers,   0.02% of elapsed)
         control-panel/fix/noexert-exhaustion       18 ms (  1.77% of lua timers,   0.01% of elapsed)
                                    husbandry       15 ms (  1.48% of lua timers,   0.01% of elapsed)
              control-panel/fix/stuck-worship       12 ms (  1.18% of lua timers,   0.01% of elapsed)
                              obsidian-caster       10 ms (  0.99% of lua timers,   0.01% of elapsed)
                                 cravingsmain        5 ms (  0.49% of lua timers,   0.00% of elapsed)
                            idle-craftingmain        4 ms (  0.39% of lua timers,   0.00% of elapsed)
         control-panel/fix/empty-wheelbarrows        3 ms (  0.30% of lua timers,   0.00% of elapsed)
                 control-panel/fix/engravings        3 ms (  0.30% of lua timers,   0.00% of elapsed)
          control-panel/fix/stuck-instruments        1 ms (  0.10% of lua timers,   0.00% of elapsed)
                                 cravingsunit        1 ms (  0.10% of lua timers,   0.00% of elapsed)

                                    framework       19 ms (  1.87% of lua timers,   0.02% of elapsed)
                                all subtimers      996 ms ( 98.13% of lua timers,   0.80% of elapsed)


Overlay details
---------------

                             gui/notify.panel     1153 ms ( 16.51% of overlay,   0.92% of elapsed)
                  agitation-rebalance.monitor      999 ms ( 14.30% of overlay,   0.80% of elapsed)
                                 hotkeys.menu      398 ms (  5.70% of overlay,   0.32% of elapsed)
                       suspendmanager.overlay      307 ms (  4.40% of overlay,   0.25% of elapsed)
                          zone.animal_actions      305 ms (  4.37% of overlay,   0.24% of elapsed)
                                  sort.places      137 ms (  1.96% of overlay,   0.11% of elapsed)
                 gui/civ-alert.big_red_button      127 ms (  1.82% of overlay,   0.10% of elapsed)
                   idle-crafting.idlecrafting       67 ms (  0.96% of overlay,   0.05% of elapsed)
                                    sort.slab       62 ms (  0.89% of overlay,   0.05% of elapsed)
                                    sort.info       57 ms (  0.82% of overlay,   0.05% of elapsed)
                                   sort.world       29 ms (  0.42% of overlay,   0.02% of elapsed)
                          sort.elevate_barony       29 ms (  0.42% of overlay,   0.02% of elapsed)
                       sort.location_selector       26 ms (  0.37% of overlay,   0.02% of elapsed)
                              sort.candidates       25 ms (  0.36% of overlay,   0.02% of elapsed)
                     orders.skillrestrictions       14 ms (  0.20% of overlay,   0.01% of elapsed)
                       buildingplan.inspector       10 ms (  0.14% of overlay,   0.01% of elapsed)
                        suspendmanager.status        2 ms (  0.03% of overlay,   0.00% of elapsed)
                        suspendmanager.toggle        1 ms (  0.01% of overlay,   0.00% of elapsed)

                                    framework     3236 ms ( 46.33% of overlay,   2.59% of elapsed)
                                all subtimers     3748 ms ( 53.67% of overlay,   3.00% of elapsed)

Note: this was over 7 days for the bad case where one of the two workshops was lacking input materials, causing it to not queue a job and run the full unit loop every 53 ticks.

Disabling the workshop that is lacking input materials, the Lua timer details look as follows:

Lua timer details
-----------------

                  control-panel/fix/ownership       77 ms ( 29.28% of lua timers,   0.07% of elapsed)
                            idle-craftingunit       30 ms ( 11.41% of lua timers,   0.03% of elapsed)
             control-panel/fix/general-strike       30 ms ( 11.41% of lua timers,   0.03% of elapsed)
              control-panel/fix/stuck-worship       23 ms (  8.75% of lua timers,   0.02% of elapsed)
         control-panel/fix/noexert-exhaustion       18 ms (  6.84% of lua timers,   0.02% of elapsed)
                            idle-craftingmain       18 ms (  6.84% of lua timers,   0.02% of elapsed)
                                    husbandry       15 ms (  5.70% of lua timers,   0.01% of elapsed)
                              obsidian-caster        9 ms (  3.42% of lua timers,   0.01% of elapsed)
                                 cravingsmain        8 ms (  3.04% of lua timers,   0.01% of elapsed)
                 control-panel/fix/engravings        8 ms (  3.04% of lua timers,   0.01% of elapsed)
         control-panel/fix/empty-wheelbarrows        1 ms (  0.38% of lua timers,   0.00% of elapsed)
          control-panel/fix/stuck-instruments        1 ms (  0.38% of lua timers,   0.00% of elapsed)
                 control-panel/fix/dead-units        1 ms (  0.38% of lua timers,   0.00% of elapsed)

                                    framework       24 ms (  9.13% of lua timers,   0.02% of elapsed)
                                all subtimers      239 ms ( 90.87% of lua timers,   0.21% of elapsed)

over the course of these 7 days, only two dwarfs were assigned to the workshop, all other iterations terminated at the very first check (i.e. that the workshop is available).

[chdoc]

So, under normal operation, the performance impact is minimal. However, de performance impact becomes noticeable if job creation fails. Maybe, it is better to remove the workshop from the configuration, if job creation fails (which should only happen if there are no materials).

[myk002]

you could also do an exponential backoff, like TCP, so it will automatically re-engage when there are materials again

[chdoc]

Hmm, exponential backoff sounds a bit annoying to implement. But it should be relatively straightforward to disable a workshop until the next run of the main loop (which runs every 8419 ticks, approximately once a week).

[myk002]

that's probably sufficient

[chdoc]

I implemented suspension of failing workshops until the next run of the main loop.

[myk002]

I suggest choosing one to be the canonical form, and only documenting that form. Both can be supported by the code, though, for flexibility.

[myk002]

Suggested change

	idle-crafting status
	idle-crafting [status]
	idle-crafting thresholds <thresholds>

[myk002]

We've discussed a little on Discord about placement. You didn't put this on the "Workers" tab because the tab is too crowded with orders overlay widgets, correct? Can we rearrange those widgets to make room for this one?

Another option is to hang the idle-crafting toggle off the side of the panel, like we do for the stocks screen:

[chdoc]

While I do think that hanging toggles of to the side is acceptable for things that are essentially full-screen, I don't think that it's the right thing to do for building cards.

The only solution I can see for gaining a reasonable amount of space would be to split the "Permitted General Work Order Labors" into two columns. However, that is a list widget, so splitting this would complicate the code elsewhere.

[myk002]

How much room does this widget need? Can it be made to fit in the area to the left of the Permitted General Work Order Labors panel?

Alternately, we could make the Permitted panel shorter and let the list be scrollable.

[chdoc]

I would say that the natural form is a single long line at the bottom. So, it technically does fit, but it does look a bit squished.

What about extending the "Permitted General Work Orders" to the full width, and moving their labels to the right of the list? This would provide the one line distance to make it not look squished, while also avoiding scrolling.

[myk002]

I think we can shrink the labor list max height by 2 rows to accommodate this. The craftsdwarf's workshop will get a scrollbar, but it's the only (vanilla) workshop that will see any change. No other workshop has such a long list.

[chdoc]

As far as I am concerned, the tool now works pretty well. However, I have only tested it using the one fort that I'm currently playing. Thus, there may (likely will) be some corner cases that I didn't anticipate. Thus, I would be happy if this tool could spend some time in testing before being released. In particular, because this should be useful for just about every fort out there. 😬

[chdoc]

Also, I am wondering whether this tool is really a "gameplay" tool, or whether it actually might qualify as "auto". Technically, you could closely watch every single dwarf, wait for a moment when they are idle, and then make them master of a workshop and queue a max-priority craft item job there. I'm not saying that this is realistic, but the tool doesn't enable any behavior that the player couldn't force using vanilla means.

[myk002]

Yeah, I had the same debate with myself about gameplay vs. auto. I settled on "gameplay" since we portrayed the tool as giving units more autonomy instead of relieving the player of a tedious burden. I can also see this getting extended later with other "abilities" that we can give dwarves.