Notebook pipeline initialization issue

[OliviaLynn]

Bug report
When I run fsps_sed_demo.ipynb, I get to this cell:

pipe.initialize(dict(input=trainFile), dict(output_dir='./temp_output_rail_fsps', log_dir='./logs_rail_fsps',
                                            resume=False, nprocess=2), None)
pipe.save('./temp_output_rail_fsps/pipe_saved.yml')

and get:

ValueError: 
Some required inputs to the pipeline could not be found,
(or possibly your pipeline is cyclic):

Stage FSPSSedModeler is missing input(s): training_data

Is this machine/environment specific, or does this happen with others?

I'd love to move the demo notebook into rail (as part of rail#48), but I think it would be better to look into this before moving it around.

Before submitting
Please check the following:

• I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
• I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
• If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

[eacharles]

This is probably a bug in how the file name generation / input alias mechanism works that was exposed by some combination of Josue and Drew’s changes a few weeks ago.  I’m traveling today.   But if you are interested in digging into this more, there were similar errors in some of the notebooks in rail during the big migration.   The exact fix depended a bit on the particulars.   In one or two cases the issue was in the notebook.   In a couple others it was a bug with how the model files were being treated.  If you have the latest versions of all the rail packages and ceci and qp and tables_io, then the issue is probably in the notebook, and in particular how the stage is being configured.

[eacharles]

This will fix your issue:

-    "pipe.initialize(dict(input=trainFile), dict(output_dir='./temp_output_rail_fsps', log_dir='./logs_rail_fsps',\n",
+    "pipe.initialize(dict(training_data=trainFile), dict(output_dir='./temp_output_rail_fsps', log_dir='./logs_rail_fsps',\n",

This in needed b/c of these lines:

training_data = DS.read_file("training_data", TableHandle, trainFile)
fspssedmodel = fspssedmodeler.fit_model(training_data)

The first line creates a DataHandle with the tag training_data, the second line tells fspssedmodeler to look for the input data under the tag training_data.

[eacharles]

Note that to get the NB to run to complete I had to make a couple other changes:

-    "fspsphotometrycreator = FSPSPhotometryCreator.make_stage(redshifts_key='redshifts',restframe_sed_key='restframe_seds',\n",
+    "fspsphotometrycreator = FSPSPhotometryCreator.make_stage(redshifts_key='redshifts',\n",
+    "                                                         redshift_key='redshifts',\n",
+    "                                                         restframe_sed_key='restframe_seds',\n",

(It seems like both redshift_key and redshift_keys are used. That should probably be fixed in the Stage)

-    "pipe.initialize(dict(input='input_galaxy_properties_fsps.h5'), dict(output_dir='./temp_output', log_dir='./logs', resume=False, nprocess=2),'./temp_output/pipe_sed_fsps_config.yml')\n",
-    "pipe.save('./temp_output/pipe_saved.yml')\n",
-    "pr = ceci.Pipeline.read('./temp_output/pipe_saved.yml')\n",
+    "pipe.initialize(dict(training_data=input_file), dict(output_dir='./temp_output_rail_fsps', log_dir='./logs', resume=False, nprocess=2),'./temp_output_rail_fsps/pipe_saved_config.yml')\n",
+    "pipe.save('./temp_output_rail_fsps/pipe_saved.yml')\n",
+    "pr = ceci.Pipeline.read('./temp_output_rail_fsps/pipe_saved.yml')\n",

(Fixing up the path)

[ztq1996]

Resolved, closing.