• Welcome to the Fantasy Writing Forums. Register Now to join us!

Independent Appraisal & Rating Service

In a different thread (somewhat derailed) it occurred to me that it would be a good idea to start up a ms appraisal and rating system (paid for) which gave an author both constructive feedback and a rating out of ten. Seven or above meant it was deemed to be of publishable quality and anything less than seven was effectively a rejection. The idea being that this was in recognition of the fact that many books rejected by mainstream publishers are of publishable quality, they simply don't fit with publishers' programs at the time.

Authors could learn from effective rejection and feedback and resubmit until they got the requisite score. This would also be a guarantee of quality to the book consuming public that any book that bore the independent rating (above seven) was worth purchasing.

Authors would submit their ms with the appraisal fee and it would be read and rated by two professional readers, both of whom would provide at least (say) 500 words on the strengths and weaknesses of the novel.

What do you think? Does anything like this already exist? Who would be interested in using the service and what should it cost?
 

buyjupiter

Maester
A quick check for "manuscript critique" brings up this through Writer's Digest: 2nd Draft Critique Service | Improve Your Writing | WritersDigestShop

My major concern would be how to do it so it wouldn't be seen as scammy. (It seems legit from what they say, but I've not used the service.) There are a lot of things out there designed to scam writers (and people) out of their hard earned money for a product that falsely proclaims success, for everyone, guaranteed.

This would, however you set it up, only work for those who are interested in growing as writers. There are some people who are more than happy to continue throwing stuff out into the market, even when they've gotten feedback it doesn't work.

I've done some critiquing (for free) elsewhere and it can get pretty ugly, even if you couch the critique in pleasant language. What provisions would you have in place in case someone thought you were maligning their darling and threatened the business or the persons conducting the business?

I've seen the going rate done by page or by hour. I think a page rate would seem fairer, $3-5 range per page? A minimum amount for a short story? Like $25 per short story or something like that?

You'd definitely want to think about how you'd define "professional reader". Is that someone with editing experience? Agent experience? Errand girl to the assistant to the assistant editor of an indie publisher?
 
Valid concerns. I'm not proposing to set this up myself, by the way, but a reputable organisation that maintained high standards would gradually acquire a reputation for non-scamminess. It would have to use numerous qualified professionals (however defined).

The price would have to reflect the service ethic of writers contributing to the community or would be otherwise beyond the means of some writers, but enough to value their time and judgment. No more than $200 total, shared between two readers with a bit for the organisation's overheads.

If you got it right it could be a great service for both writers and readers.
 

buyjupiter

Maester
I really wish there were a separate form for the plural "you" (I'm trying to cut "y'all" out of my writing and "one" is too formal). I didn't figure you(singular) wanted to take on the challenge of starting a business like this.

But this is something the freelance editors on the forum might want to consider adding to their services.
 

Philip Overby

Staff
Article Team
I think this is a good idea, but of course anything like that always comes with some caveats.

1. What determines a professional reader? Someone who has read and critiqued a lot of stories? I guess the issue may be finding people who seem qualified enough to dispense advice to the point that it would be worth money.

2. There are already critiquing services out there that are free like Critters. Some people don't like their system, but they seem pretty successful. What would warrant charging for the reading service? I mean what would this service offer that others aren't?

3. I'd say things like this tend to involve writers critiquing other writers. Meaning a lot of times the person critiquing may not be able to read something strictly for entertainment or marketing value, which might be what a lot of writers would be looking for. Not to say writers critiquing writers is necessarily bad, it's very helpful obviously, but I can get that for free by trading critiques with someone I trust.

All that said, perhaps this proposed service would be most helpful for those that don't have time to trade critiques and just want some feedback before sending their work out into the world. Perhaps having a pool of readers who could be rated by the clients, much like freelance services function. Therefore if you're ranked 1 on a scale of 1-5, then you might be cheaper, but your reading is not as thorough. If you're ranked 5, then the writer submitting their manuscript might expect detailed beta reader notes. In this case, the clients and the service could help rank the feedback that they're receiving. Then if a reader says, "Yeah, I liked it" they'd probably get ranked 1 and not many people would use them. But if the reader gives detailed notes and spends time on it, then they may get ranked 4 or 5 and would be recommended by other writers.

I'm sure there would be some willing to submit to a service like this if it meant they could get tons of useful feedback in a short amount of time. I mean, me for instance, when I did some freelance writing, I was paid for the amount of words I wrote and then I had a deadline. If the freelance readers were given a deadline to get their comments in in order to get paid, it might increase the productivity. Money always speeds things along.

So that said, I think this is a good concept, but would need to be executed by the right people who not only have the reputation (like a Writer's Digest or some such), but also have the business acumen to pull off something like this without getting into messy lawsuits.

I also think Dark One's original proposal sounds good, in which authors strive to get their work to a certain rating level as well. That would make writers be more careful before sending their work to be published or self-published.
 
Last edited:

PaulineMRoss

Inkling
I don't see any objective way to rate a book on a 1-10 scale. You can say: yes, it meets a minimum standard for grammar, spelling and punctuation. That's quantifiable.

But having engaging characters? Having a sensible plot? Having credible motivations for characters' actions? These are very subjective. Something that one reader loves as a pacy, action-packed adventure may be another person's over the top explosion-fest with cardboard characters. What might be a character-driven, introspective and thoughtful read to one person is a whiny, angst-ridden pile of boredom to another. You can't reduce all of that to a single number and say: this one is 'publishable' and that one isn't. Whatever 'publishable' means.
 

C Hollis

Troubadour
In my mind, the pro readers would have to be former agents, professional editors, etc. People with credentials.

As far as the benefit over your standard critique from fellow writers, this is where I may been seen as a bit harsh.
Publishing is a business, and a setup like this should be treated as such. You're not submitting your work to your mother to get her stamp of approval, you are submitting your work to get (looking forward here) an industry recognized rating that must maintain its own integrity. Therefore, you won't get the have-to-sugarcoat-to-not-discourage critique, you will get the hardcore no-nonsense truth as they see it.
As I mentioned in the other thread, whether the manuscript is salable on a grand scale shouldn't even be a consideration with this service. Is it a well-written, complete (beginning, muddle, end) story?

I think the biggest hurdle, for the service, would be structuring the reviews and process in a manner where personal opinions are muted (won't be able to eliminate them). You don't want writers to start saying things like "Well, Jane hates sci-fi, so she always gives low ratings to those stories. Re-submit, maybe Bill will get the story. He's a sci-fi geek."
 

Philip Overby

Staff
Article Team
More good points made. These kind of things always have lots of issues to consider. I'm almost thinking doing something like oDesk or some of the bidding sites would be a good alternative. In this case, a site that focuses on fiction writing first and foremost. A network where you can find editors, cover artists, and potentially paid readers. oDesk often allows people to bid on a job by stating their qualifications, writing a cover letter, etc. In the case of a reader service, the writer would have several potential readers bid on the chance to read the novel. The writer would then chose who they wanted based on his or her skills, interests, etc. This could avoid the "so and so doesn't like action novels" and such. However, some don't like the bidding model. In this case, I think that might work.

When it comes to the need for a service like this, I definitely think it would be viable. I constantly see people looking for beta readers, but it always comes with the catch, "read mine and I'll read yours." While this is good in principle, some writers may find the time management spent reading dozens of other people's work just so they'll read theirs might not be the best thing for them.
 

PaulineMRoss

Inkling
As far as the benefit over your standard critique from fellow writers, this is where I may been seen as a bit harsh. [...] Therefore, you won't get the have-to-sugarcoat-to-not-discourage critique, you will get the hardcore no-nonsense truth as they see it.

Honestly, if a critique group is sugarcoating, it's time to find a new critique group. If a writer is just looking for someone to make encouraging noises, they're not ready for the real world. Any critique should be hardcore no-nonsense truth (which includes saying what works as well as what doesn't, of course).

Otherwise, your points are well-made. I just have trouble imagining any system which can provide an objective assessment of quality in writing. But I guess that's just me.
 

BWFoster78

Myth Weaver
Dark One,

I think the service you've brought up is definitely needed. There are a lot of authors out there like me who are thinking, "As far as I can tell, my work is up to snuff, but how can I be sure?"

I'd take the service a step farther, though.

Frankly, it's difficult to find good indie books. The service could make extra money by pointing readers to the best books. That also would make it more valuable for the author as they not only would get validation but marketing as well.
 

BWFoster78

Myth Weaver
I don't see any objective way to rate a book on a 1-10 scale. You can say: yes, it meets a minimum standard for grammar, spelling and punctuation. That's quantifiable.

But having engaging characters? Having a sensible plot? Having credible motivations for characters' actions? These are very subjective. Something that one reader loves as a pacy, action-packed adventure may be another person's over the top explosion-fest with cardboard characters. What might be a character-driven, introspective and thoughtful read to one person is a whiny, angst-ridden pile of boredom to another. You can't reduce all of that to a single number and say: this one is 'publishable' and that one isn't. Whatever 'publishable' means.

I'm not sure I agree with you here.

I'm reading an indie book now that I'm almost finished with. Frankly, I'm not enjoying it that much. However, I can tell you that the quality of the book is just fine; it's just that I don't particularly care for the story and characters.

The point is that a reader can objectively say, "Even though I didn't care for this book, the quality level is acceptable."

No one is going to like every book no matter how well done it is. This service wouldn't be a "review" but a stamp of quality. The point isn't to tell the reader, "Hey, you'll like this book." It would be to tell the reader, "Hey, this author actually knows how to write. Check out the reviews and see if the characters and story will appeal to you."
 

PaulineMRoss

Inkling
The point is that a reader can objectively say, "Even though I didn't care for this book, the quality level is acceptable."

Only if you can set a sensible measure for an acceptable quality level. Beyond the absolute minimum (correct grammar, punctuation and spelling), what determines that a book is acceptable? Someone upthread said something about having a beginning, middle and end, but that's pretty basic, too. And anything beyond that - relateable characters, a plot that makes sense, good description - is subjective, I would assert.

Don't get me wrong, I think it's a great idea, in principle, I just don't see how it would work in practice.
 

BWFoster78

Myth Weaver
Only if you can set a sensible measure for an acceptable quality level.

What's the famous quote about pornography? Something like, "I can't define it, but I know what it is when I see it."

I kinda think it's the same for writing. Even if I don't like a book, I can tell you whether or not the writing is of a sufficient quality, and I think I can do that even without a definitive measure of what that acceptable quality level is.

As a practical matter, maybe you take a few really well written short pieces as examples and say, "There are in the 8-10 range." Then, take some decent ones and say, "These are in the 4-7 range." Then, find some really poorly written crapola and say, "These are in the 1-3 range." Every evaluator would be required to read the example pieces to get an idea of the quality.

Writing is so subjective that I'm not sure you'd ever get an acceptable definitive standard established.

Maybe, again as a practical matter, new evaluators go through an "training" period where their rankings are looked over by an experienced evaluator.

I do think that, since it's so subjective, perhaps The Dark Ones 10 point scale wouldn't work. Maybe a "Terrible," "Acceptable," and "Good" system would be better.
 
Just to respond quickly to Pauline's question about rating, rating/evaluating books is what readers and publishers do all the time. Maybe not in some crass numerical way (despite the hugeness of goodreads) but they still do it in ways meaningful to themselves. That's why the proposed system would always use two reviewers to mitigate the taste issue but also to give confidence that at least two reviewers thought the book publishable (or not).

You could play about with the grading system, give guidance and moderate, just like any grading system at university.

The thing is we need an independent rating system which is valuable and affordable - giving advice to the 'rejected' and marketable oomph to the validated. The right people involved at the right price, doing something like what is proposed would be doing a great service to the independent publishing business.

They could even set up their own imprint. Independent Publishers Co-op? It would rival mainstream publishers because they are nothing more than filters in the giant ocean of online crap...just like this would be.
 
I certainly applaud the concept and see the need for such a thing, but I'm firmly with PaulineMRoss that there is no practical way to implement such a thing. Adherence to grammar is about the only thing you can attempt to judge but there is a world of difference between a grammatically correct book and one that weaves a compelling tale. Then there are those who violate the rules of grammar for dramatic effect. Cormac McCarthy's books would likely fail a "grammar test" given his use of fragments, lack of punctuation and run-on sentences, but given his Pulitzer Award and sales numbers the fact that he would score below the threshold really doesn't hold water.

With a single individual they can establish a criteria by which they judge books...and then they can rank the books as such. Each of us has this ability based on our subjective preferences, but once we try to expand that beyond the individual everything falls apart.

I've sold almost half a million English language books, and have thousands of positive reviews, but I...like everyone in the business...have the 1-star reviews that are completely baffled by how such a book could ever be published, let alone thought well of by others. Even if populated by "professionals" like editors, publishers, and agents...consider how many books were rejected multiple times as "unworthy" and yet once they got out in front of people, they sold well and were loved by readers?

Again I concur with the desire to fix something - I just don't think we can have any system that "grades" books considering the complexity and diversity of what makes a book "good."
 

danr62

Sage
I'm with Pauline and Michael here. There just isn't a good, objective way to do this.

Not only that, but trad authors have agents and editors to tell them if their books are publishable quaility or not, and self published authors, largely, wouldn't spend the money on this.
 
Even if populated by "professionals" like editors, publishers, and agents...consider how many books were rejected multiple times as "unworthy" and yet once they got out in front of people, they sold well and were loved by readers?

Again I concur with the desire to fix something - I just don't think we can have any system that "grades" books considering the complexity and diversity of what makes a book "good."

Let's not forget that those rejections occurred in the context of the current paradigm - ie, not many publishing slots available. The paradigm we're talking about is not limited by available slots, it is only limited by quality (however defined).

When I used to do slush pile reading, I wasn't reading for slots, just for publishable quality. I didn't have any training - just the confidence of the publisher. I think everyone would have their own idea of what is publishable and you might be surprised how easy it is to make a judgment in an area you felt competent to do so...after a bit of practice.

Anyway, I'm not proposing we do it. I'm proposing someone with decent credentials do it but unconstrained by the usual scarce publishing slots.
 

Philip Overby

Staff
Article Team
I do think one thing that attracts self-publishers is not having to deal with gatekeepers to get their work published. A service like this would be most beneficial to readers looking for quality indie fiction.

I've long thought a service that simulated a traditional publisher could be beneficial in some way for writers not sure if their writing is up to snuff yet. Maybe it could have like a group of fake publishers (maybe industry professionals, volunteers, etc.) who treat your manuscript as a traditional publisher would. Like they wouldn't publish anything, but still put the same level of scrutiny that a traditional publisher might in regards to their personal feelings about submissions. So even if the manuscript is "rejected" there would be other "simulated publishers" that you could submit your work to. Even if you disagree with their ideas, you at least got some feedback and could still self-publish your work either using the feedback or not. If your manuscript gets "accepted" perhaps the fake publishers could offer up some rewards for those authors by way of promotion of their self-published book, free cover art, reviews, etc. The "accepted" self-published books might get some kind of stamp of approval from the service. Something like this might be best done through some kind of start-up like Indiegogo or Kickstarter.

Of course this might just be a really complicated way of me saying, "get really thorough beta readers." :)
 
Last edited:

Caged Maiden

Staff
Article Team
the hardest part of this for me is a star rating of sorts. If I were going to self-publish, I'd certainly consider doing the equivalent of Paypal Verifying, sending my manuscript to an independent service to be read and "approved" for publishing. However, I'd want a detailed reason for the grade, whether it was ten or five or one. 500 words wouldn't do it for me. I know it takes a long time to read someone's novel... more if it's terrible... but the grading system should probably have an attached worksheet to briefly hit on all points.

If my book scored a 7, I wouldn't know whether that meant it was good but needed improvement or what. Would 500 words possibly indicate that everything was great, apart from stilted dialogue throughout most of the story? Notes would be really important.

Also, I see anther potential pitfall... I have had a lot of people beta read my WiP. Some really enjoy my MC, a girl who is very naive. Mostly men really seem to enjoy her parts of the story and they tell me so. But women tend to hate this character. Two have stopped reading because of her. Wow. Since women are my target audience... what do I do? How could a service using only two readers, with 500 words of information, possibly deal with a problem like that? Perfect grammar aside, I could get failed for the sake of turning off the reader.

Concerns aside, I'd still use such a service if it existed. I look at the number of other businesses that have purchased the best payment gateway I found (Authorize.net) and I recognize their stamps on every website I pass. Once a badge is popularized, you sort of need it. I'd certainly think that if such a rating system occurred, giving a book a stamp of approval, writers would seek it out (at least those who were serious). It would be something that would be copy-catted and probably improved upon, though.

Also... let's face it $200 is nothing for said service. With two people reading say, a 100k word novel... they are not making enough, unless maybe we're talking about skimming more than reading? I can't really do that, so I'm not sure how it works. Maybe you could crit the first three chapters like an agent would and either "request a partial" or "reject" the manuscript?
 

TWErvin2

Auror
One way I could see this possibly working is some independent organization could read a published work (at the request of the self-published author, probably for a fee) and if it met minimum standards:

Spelling/Grammar/Punctuation etc.
No Major Plot Holes

Adequate Dialogue
Storyline that goes somewhere

Maybe a 'stamp' of approval could be earned by the book by the organization. However, the two latter items that I listed, for example, would be subjective. It's much more difficult when one delves into subjective areas.

It would be even more effective if a stamp of disapproval would be included with those novels that did not make the cut.

Again, the subjective. There are some reviewers who are good at what they do, but I rarely agree with their opinion on a novel.
 
Top