PULP_CBC_CMD not working with P-Median/P-Center

[mbforr]

For some reason it is working with LSCP and MCLP but not P-Median or P-Center and it is just giving the generic error back:

Pulp: Error while executing /usr/local/lib/python3.9/dist-packages/pulp/solverdir/cbc/linux/64/cbc

I have tried several different recommendations from the pulp docs and nothing worked there. Tested this out in Colab, a local notebook running on Docker, and in a plain Linux env. Tried it with the pulp install inside spopt and when installing it separately.

[gegen07]

Hey @mbforr!
Are you using the spopt example notebooks?

By the way, check if PULP_CBC_CMD is available in your environment with pulp.listSolvers(onlyAvailable=True).

[mbforr]

No I'm using my own since I am using a cost matrix that originated from pgRouting as a CSV which is structured into the correct 2D array based on the examples. And yes when I run that code PULP_CBC_CMD is available

[jGaboardi]

We (read that as I) desperately need get around to #348...

@mbforr Have a look here --> #356 (comment)

[mbforr]

So I found those issues and used that code but had the same issue. P-Center/P-Median don't work with those but it does with LSCP and MCLP.

[jGaboardi]

OK, we'll need some more information to attempt to properly troubleshoot this issue. Can you provide/answer the following:

• The version of spopt and pulp that you are running.
• How did you install spopt and pulp?
• A minimum reproducible example of code.
• The full traceback for the error.
• Do the $p$-median and $p$-center notebooks run successfully without any change (with the example data provided)?
• Can you provide you example data?

@gegen07 Can you think of anything else?

[mbforr]

Sure thing!

1. spopt 0.5.0 and pulp 2.7.0
2. I only installed spopt through pip install spopt. I tried this in the context of the notebook and within the base Docker container I am using locally
3. See notebook here - https://gist.github.com/mbforr/71a8a89bfbb2205a75c053b2519ba298
4. Error is below
5. Testing those now :)
6. Data is attached here odm.csv

PulpSolverError                           Traceback (most recent call last)
[<ipython-input-8-e563971d49ab>](https://localhost:8080/#) in <cell line: 1>()
----> 1 pcenter_from_cost_matrix = pcenter_from_cost_matrix.solve(
      2     pulp.PULP_CBC_CMD(msg=True)
      3 )

3 frames
[/usr/local/lib/python3.10/dist-packages/spopt/locate/p_center.py](https://localhost:8080/#) in solve(self, solver, results)
    407 
    408         """
--> 409         self.problem.solve(solver)
    410         self.check_status()
    411 

[/usr/local/lib/python3.10/dist-packages/pulp/pulp.py](https://localhost:8080/#) in solve(self, solver, **kwargs)
   1911         # time it
   1912         self.startClock()
-> 1913         status = solver.actualSolve(self, **kwargs)
   1914         self.stopClock()
   1915         self.restoreObjective(wasNone, dummyVar)

[/usr/local/lib/python3.10/dist-packages/pulp/apis/coin_api.py](https://localhost:8080/#) in actualSolve(self, lp, **kwargs)
    135     def actualSolve(self, lp, **kwargs):
    136         """Solve a well formulated lp problem"""
--> 137         return self.solve_CBC(lp, **kwargs)
    138 
    139     def available(self):

[/usr/local/lib/python3.10/dist-packages/pulp/apis/coin_api.py](https://localhost:8080/#) in solve_CBC(self, lp, use_mps)
    211             pipe.close()
    212         if not os.path.exists(tmpSol):
--> 213             raise PulpSolverError("Pulp: Error while executing " + self.path)
    214         (
    215             status,

PulpSolverError: Pulp: Error while executing /usr/local/lib/python3.10/dist-packages/pulp/solverdir/cbc/linux/64/cbc```

[jGaboardi]

@mbforr You are having this issue because there are nan values in the array (d) you are passing in, specifically at the following indices:

>>> numpy.where(numpy.isnan(d))
(array([ 564,  911, 1633, 2034, 2372]), array([3, 4, 6, 7, 8]))
# for example
>>> d[564, 3]
nan

When running cell 8 of your example notebook I get the following message from CBC:

Welcome to the CBC MILP Solver 
Version: 2.10.3 
Build Date: Dec 15 2019 

command line - /Users/user/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/solverdir/cbc/osx/64/cbc /var/folders/71/bk36jbtj26n_v5fhw9tnzzl00000gn/T/d689d8e073714c70bca0043ca1d60209-pulp.mps timeMode elapsed branch printingOptions all solution /var/folders/71/bk36jbtj26n_v5fhw9tnzzl00000gn/T/d689d8e073714c70bca0043ca1d60209-pulp.sol (default strategy 1)
At line 2 NAME          MODEL
At line 3 ROWS
At line 26615 COLUMNS
Bad image at line 82502 <     X0006343  C0025824   nan >
Bad image at line 102577 <     X0010358  C0026225   nan >
Bad image at line 119457 <     X0013734  C0026563   nan >
Bad image at line 137882 <     X0017419  C0024755   nan >
Bad image at line 155257 <     X0020894  C0025102   nan >
At line 159689 RHS
Bad image at line 184445 <     RHS       C0024755   nan >
Bad image at line 184792 <     RHS       C0025102   nan >
Bad image at line 185514 <     RHS       C0025824   nan >
Bad image at line 185915 <     RHS       C0026225   nan >
Bad image at line 186253 <     RHS       C0026563   nan >
At line 186300 BOUNDS
At line 208081 ENDATA
Problem MODEL has 26610 rows, 21781 columns and 89507 elements
Coin0008I MODEL read with 10 errors
There were 10 errors on input
Option for timeMode changed from cpu to elapsed
** Current model not valid
Option for printingOptions changed from normal to all
** Current model not valid
No match for /var/folders/71/bk36jbtj26n_v5fhw9tnzzl00000gn/T/d689d8e073714c70bca0043ca1d60209-pulp.sol - ? for list of commands
Total time (CPU seconds):       0.05   (Wallclock seconds):       0.05

---------------------------------------------------------------------------
PulpSolverError                           Traceback (most recent call last)
Cell In[8], line 1
----> 1 pcenter_from_cost_matrix = pcenter_from_cost_matrix.solve(
      2     pulp.PULP_CBC_CMD(msg=True)
      3 )

File ~/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/spopt/locate/p_center.py:409, in PCenter.solve(self, solver, results)
    390 def solve(self, solver: pulp.LpSolver, results: bool = True):
    391     """
    392     Solve the ``PCenter`` model.
    393 
   (...)
    407 
    408     """
--> 409     self.problem.solve(solver)
    410     self.check_status()
    412     if results:

File ~/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/pulp.py:1913, in LpProblem.solve(self, solver, **kwargs)
   1911 # time it
   1912 self.startClock()
-> 1913 status = solver.actualSolve(self, **kwargs)
   1914 self.stopClock()
   1915 self.restoreObjective(wasNone, dummyVar)

File ~/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/apis/coin_api.py:137, in COIN_CMD.actualSolve(self, lp, **kwargs)
    135 def actualSolve(self, lp, **kwargs):
    136     """Solve a well formulated lp problem"""
--> 137     return self.solve_CBC(lp, **kwargs)

File ~/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/apis/coin_api.py:213, in COIN_CMD.solve_CBC(self, lp, use_mps)
    211     pipe.close()
    212 if not os.path.exists(tmpSol):
--> 213     raise PulpSolverError("Pulp: Error while executing " + self.path)
    214 (
    215     status,
    216     values,
   (...)
    220     sol_status,
    221 ) = self.readsol_MPS(tmpSol, lp, vs, variablesNames, constraintsNames)
    222 lp.assignVarsVals(values)

PulpSolverError: Pulp: Error while executing /Users/user/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/solverdir/cbc/osx/64/cbc

You can see from above that there are nan values in the LP formulation, which is not allowed (xref coin-or/pulp#384 (reply in thread)).

After changing those nan values to a large number (I used 1,000,000 – inf is also no allowed), the p-center model solves in ≈2.5 seconds on my machine.

Welcome to the CBC MILP Solver 
Version: 2.10.3 
Build Date: Dec 15 2019 

command line - /Users/user/mambaforge/envs/forrest_pulp_spopt/lib/python3.11/site-packages/pulp/solverdir/cbc/osx/64/cbc /var/folders/71/bk36jbtj26n_v5fhw9tnzzl00000gn/T/46babba155894930816da8d52ed5d767-pulp.mps timeMode elapsed branch printingOptions all solution /var/folders/71/bk36jbtj26n_v5fhw9tnzzl00000gn/T/46babba155894930816da8d52ed5d767-pulp.sol (default strategy 1)
At line 2 NAME          MODEL
At line 3 ROWS
At line 26615 COLUMNS
At line 159689 RHS
At line 186300 BOUNDS
At line 208081 ENDATA
Problem MODEL has 26610 rows, 21781 columns and 89512 elements
Coin0008I MODEL read with 0 errors
Option for timeMode changed from cpu to elapsed
Continuous objective value is 146.46 - 0.36 seconds
Cgl0004I processed model has 26610 rows, 21781 columns (21780 integer (21780 of which binary)) and 89512 elements
Cbc0038I Initial state - 12 integers unsatisfied sum - 1.86724
Cbc0038I Pass   1: (2.25 seconds) suminf.    0.00000 (0) obj. 157.09 iterations 2960
Cbc0038I Solution found of 157.09
Cbc0038I Relaxing continuous gives 157.09
Cbc0038I Before mini branch and bound, 21768 integers at bound fixed and 0 continuous
Cbc0038I Full problem 26610 rows 21781 columns, reduced to 0 rows 0 columns
Cbc0038I Mini branch and bound improved solution from 157.09 to 146.46 (2.35 seconds)
Cbc0038I After 2.35 seconds - Feasibility pump exiting with objective of 146.46 - took 0.21 seconds
Cbc0012I Integer solution of 146.46 found by feasibility pump after 0 iterations and 0 nodes (2.36 seconds)
Cbc0001I Search completed - best objective 146.46, took 0 iterations and 0 nodes (2.38 seconds)
Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
Cuts at root node changed objective from 146.46 to 146.46
Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

Result - Optimal solution found

Objective value:                146.46000000
Enumerated nodes:               0
Total iterations:               0
Time (CPU seconds):             2.33
Time (Wallclock seconds):       2.46

Option for printingOptions changed from normal to all
Total time (CPU seconds):       2.41   (Wallclock seconds):       2.54

[jGaboardi]

@mbforr One thing you may consider is using the .pivot_table() method instead of .pivot() on your dataframe. This will allow for using the fill_value keyword, which replaces missing values (the origin == destination in that data, e.g. location 783340 that is in both "start_vid" and "end_vid").

I think this solves your problem, so I will go ahead and close the issue. We can reopen if you are still having trouble.

[mbforr]

thanks! i checked the matrix with isnull in pandas prior to the pivot but i think that the pivot method is right. sorry about that and thanks for the tip!