Addition to last commit b404c3af to do the same thing for removing.

Also a very minor sanity check in job_mgr.c to make sure we at least have
a task count.  This shouldn't matter, but just to be as robust as possible.