Note 419-4 — Scene I.—Before Prospero’s Cell.
Reading edition
The meaning of this line probably is that when he works the least he is really most wearied because he does not have Miranda’s sympathetic words to cheer him, or the sweet thought that he is working for her.