To answer this question, you first have to determine what your goals are for the outcome of the editing process.
The way I see it, the primary goals of any editing session are to make the manuscript:
Professionally polished (for the benefit of agents and other discerning readers);
Entertaining and possibly educational (for the benefit of paying customers and/or reviewers).
From these two goals flow all other hallmarks of success: a good sales rate, overwhelmingly positive reviews, and so on.
To achieve one or both of these goals, you should focus on the following editorial concerns:
Structural issues like pacing;
Development of engaging, sympathetic characters whose actions and choices are easy for the reader to buy into.
I talk about both of these in my Top 5 Editorial Mistakes… blog series and then I go really in depth, get really nitty-gritty in my forthcoming course on getting organized to edit well. (I’m really freaking excited to finally release this to you guys soon. Not even joking. Learn more about it here.)
Oh, and you should avoid grammar and spelling issues whenever and wherever possible. Readers looove to email authors about spelling mistakes. Even if you don’t have the funds to hire a developmental editor, be sure to hire a copyeditor, a proofreader, or both.
A good rule of thumb to know you’re done editing is that you’ve revised your manuscript at least twice.
That’s at least once based on your own assessment of what needs work and at least once after that based on the assessment of someone else, whether a professional editor or a trusted critique partner.
Then, listen out for the still, small voice that tells you the book is ready. You might call it intuition, Spirit, whatever. For me, it feels like a cool, smooth stone settling into the bottom of my stomach. For you, it might feel different or less woo-woo, but it should be, for lack of a better term, a “gut feeling,” one that is stronger or at least more persistent than the many anxious voices flooding your head. (You know what I’m talking about: “What if this isn’t right? What if that isn’t? What about that comma on page 236?!”)
Anecdotally, I’ve heard published authors say that even when their book is out on the market they feel like there are things left to play with. Maybe there are, maybe there aren’t. (For instance, I agonized over whether to use the more proper semicolon to break up that last sentence, but ultimately went with a comma, as it reads a little more easily on the eyes.) At the end of the day, you can always revise and release a new edition down the road.
Bottom line: You’ll never be fully done editing, but there’s polished and there’s unpolished.
Be sure you’re tackling structural issues like pacing and developing sympathetic characters and smoothing minor wrinkles like spelling errors. Beyond that, be true to yourself. When you know at the pit of your stomach that the book is ready—probably around the same time you start fretting and googling this question, then reading my blog along with seventeen other editors’—then you’re ready to send that polished manuscript out into the world.