(Not intended to be exhaustive. ?)

On failures:

• Why did my job fail?
  • Ok, I saw the error code, but what did that actually mean?
  • Can you make changes on the cluster itself so this will succeed next time?
• What physical node did my job run on?
  • How can I make sure my retries don’t run on that node again?
  • Ok, I understand that the machines are identical, but can I just make sure to never run on that node ever again?
• What can I do to prevent this failure from happening again?

On priorities or “quality of service” mechanisms:

• Why is my job taking so long to run? (And/or, why are my requests being throttled?)
• How can I increase my priority to avoid this throttling again?
• Why is that other user’s job running with a higher priority of mine?
  • Can you please throttle their jobs which are getting an unfair priority?
• I have a critical deadline! Can you please override the priority score so my job runs immediately?
• Can you reserve some set of resources for my dedicated use for some amount of time?
• Can you provide some guarantee that my jobs will always run within a specified time?
• Can I get an interactive way to run on the machine? I don’t want to deal with writing a job script.
  • (This may be requested for a single node, or at any scale up to the full cluster!)

On the development environment:

• Can you please provide package X on the system? (Where X may be something the admin team has never heard of)
• Why doesn’t the cluster provide the newest version of X? (Or an updated version of some API)
  • When can I get access to the newest version?
• Why did you upgrade the cluster so quick? Now my workflow is broken.
  • Can you please go back to the version that works for my job?

---

Now, I’ll grant you that I can get a little snarky sometimes, but all of these questions may have some valid business reason!

Even seemingly obnoxious requests, like the user who shows up on a Friday asking for exclusive access to the full cluster, might actually turn out to be the most important thing to do in that context.

And valid or not — most of these are questions the users of any shared resource will eventually ask! I’ve run across too many cluster stacks that can’t actually inspect their priority system; don’t provide any tracing for failures; or can’t even tell the user which machines they ran on.

One way or another, you should probably be able to answer these kinds of questions, or you’ll have a lot of trouble operating your system over time.