• Welcome to the Fantasy Writing Forums. Register Now to join us!

Independent Appraisal & Rating Service

T.Allen.Smith

Staff
Moderator
A way to minimize subjectivity would be increasing the sample size. Most reviews are done by one person, and therefore very subjective.

However, if a rating service (not a review service) employed a team of 20+ readers, and each reader independently graded the work on a scale of 1-5, the average would be a better representation of differing likes and dislikes.

There would be no review. Simply a standard form for each reader to fill out at completion. Grades are compiled and averaged...done. Your book received a rating from "Such & Such Reader Group" of 3.8 stars....
 

TWErvin2

Auror
However, if a rating service (not a review service) employed a team of 20+ readers, and each reader independently graded the work on a scale of 1-5, the average would be a better representation of differing likes and dislikes.
That would make it unfeasible...20 readers per title.

If a project opened with 1000 readers, and each reader could read 10 titles per week (2 per day), that would be: 26,000 titles per year. That's a drop in the ever growing self-publishing bucket...and that's with 1000 hired readers.

Then, with 20 readers for each book, what reasonable fee would be charged for the service? If a reader read 400 words per minute, a 80,000 word novel would take 3.33 hours. Say 4 hours per book after completing the form and a 15 minute break (thus in an 8 hour day two books could be read and evaluated). If paid roughly minimum wage (USA) at $8.00/hour, that would mean $640 to just pay the readers, plus overhead and paying managers, say $700 per book. I believe that fee for the service would be unlikely to attract many self-published authors. Since it's skilled labor, to an extent, $12.00/hour might be more reasonable, so that would be over $1000 to have the book read and properly rated.

Maybe if it could get off the ground and prove to be an asset to self-published authors, it might work, but to get that level of notoriety/confidence by readership, it would could easily take two years and quite a bit of advertisement/PR. Those costs would have to be recouped in addition to getting authors to pay up initially to get the process going for two years.

Or it would take someone with very deep pockets and a good vision (or some serious venture capital or a very impressive kickstarter campaign).
 

T.Allen.Smith

Staff
Moderator
I don't disagree Terry. Everything you say is valid. Still, the only way to reduce subjectivity is to increase the sample size in any rating system.
 
The whole point of paying for the service is that you are paying for professional expertise in lieu of numerically valid objectivity. Simply wouldn't be feasible otherwise, and it would be up to the raters to acquire legitimacy and standing.
 
Hi,

I agree the service is needed, but there's simply no way it's going to happen any time soon. There are just too many hurdles.

First there are over a million indie books on Amazon right now and more every day. If even one percent of authors saw this as something they'd want, what sort of group could meet the demand? Who has the resources?

Second, there's the question of indipendence. Said vetting body would have to be completely indipendent, and on top of that professional. So that means no authors, no publishers, no agents and not even fora like SFWA. Most of the industry would instantly be ruled out as potentially biased.

Then there's name brand issue. What does it mean to a reader to see that a book is vetted as four stars by Joe Blogs? They have to know the name or it means nothing.

After that there's the question of what the author does with the vet. He wants to use it assuming it's good, but Amazon won't allow any reviews that have been paid for. They also disallow loads of others for various reasons. So a book could be vetted and it still be completely useless to the author.

Then there's subjectivity. As has been said, sometimes even the rules of grammar are broken deliberately by authors. Can you really grade a book down for that? As for plot and character etc, what one person likes another hates. The whole thing becomes completely subjective.

There are just too many hurdles in my view. It's why I said it wouldn't happen in 2014 in my predictions blog.

Cheers, Greg.
 

Devor

Fiery Keeper of the Hat
Moderator
In a different thread (somewhat derailed) it occurred to me that it would be a good idea to start up a ms appraisal and rating system (paid for) which gave an author both constructive feedback and a rating out of ten.

So are you paying for the edit, the review or the approval? I think it would be better to separate the services.

What the industry needs, I think, is a stamp (or, y'know, several competing stamps would do). The stamp would simply mean:

"Mythic Scribes (or whoever) deems that this book is up to traditionally published standards."

And that's it. No critiques. No rating. No reviews. No feedback. Submit the book, pay your fee, and either you get the stamp or you don't. Thanks for submitting. If you're rejected you can resubmit in six months after editing. No hassles.

Because I don't need help picking out which self-published book to read. I need help picking out which books are readable. Get people over that hurdle, and you're gold.

Done right that one thing might fix the whole industry.
 
Last edited:

PaulineMRoss

Inkling
What the industry needs, I think, is a stamp (or, y'know, several competing stamps would do). The stamp would simply mean:
"Mythic Scribes (or whoever) deems that this book is up to traditionally published standards."
And that's it. No critiques. No rating. No reviews. No feedback. Submit the book, pay your fee, and either you get the stamp or you don't.

There already are people who will do that. Pay a fee, get a stamp (or not). And do I trust them? Of course not. Anything the author has paid for is suspect. Plus, the ones I've seen have given stamps to books that I don't think are up to standard.

Whatever system evolves, it has to be independent of anyone with a vested interest in the books concerned. Goodreads is the nearest thing to independent, but it's deeply flawed too. Maybe we need a Rotten Tomatoes for books.
 

Devor

Fiery Keeper of the Hat
Moderator
There already are people who will do that. Pay a fee, get a stamp (or not). And do I trust them? Of course not. Anything the author has paid for is suspect. Plus, the ones I've seen have given stamps to books that I don't think are up to standard.

Whatever system evolves, it has to be independent of anyone with a vested interest in the books concerned. Goodreads is the nearest thing to independent, but it's deeply flawed too. Maybe we need a Rotten Tomatoes for books.

There's only two ways to do it. Either you find a way to make money doing it, or you crowdsource it to the public. There's too many books to do anything else. But any time you outsource it to the community, you let in too many variables and unknowns for quality control. At least when there's money involved there are ways to earn trust and build a reputation over time.
 
Hi,

Had a small brainwave about this, and have already suggested it to Amazon and put it out for comment on Kindle Boards.

A seal of approval is if not impossible the next thing to it. There are just too many hurdles in my view. But a seal of sales is a different thing.

Now here I'm thinking of the music industry, specifically gold records and platinum records. Now everyone knows that gold records and platinum records relate to sales. But they are also perceived rightly or wrongly as a stamp of apporval of some sort. They tell buyers, x thousand people have bought this record. And buyers see this and say well if x thousand people have bought this it can't be too terrible. I mean the number of times I've heard people arguing about songs and records and one guy says it's crap and the other says well it went double platinum etc in responce is ridiculous.

You can do the same thing with books. So here I'd be thinking golden quills(?) Say your book gets a silver for a thousand sales. A gold for ten thousand. Platinum for a hundred thousand. Amazon processes the sales and when a book achieves the requisite number the quill is automatically stamped on the book page. And then readers see this, and automatically know that this book has sold a certain number of sales.

That's prestige for the author (something he can take to agents, publishers and professional author bodies if he chooses to), a sort of quality stamp for the book, and comfort for the readers. It's also an obvious marketing tool.

And in terms of the quality issue, it does add to that, for the same reason that the quill brings a measure of comfort to readers. The theory is that if a book has sold a thousand copies it must have some basic qualty. While those that don't have that sort of quality should never achieve it. People presumably won't buy them.

This works a little better than a sales rank, since ranks go up and down, and total sales just rise, slowly or quickly. Also I suspect most readers don't even notice a book's sales rank.

My main proviso would be that it could only be for books actually sold (no freebies since people take anything free).

What do you guys think? Does it answer some of the issues raised?

Cheers, Greg.
 
Answers some questions but not the main one for me...I was talking about a badge of quality that assures a purchaser that they're buying something written and edited to publishable quality (in the opinion of the raters). Your system is arguably indicative of quality but is also just a popularity contest. What's to stop an author buying a 1000 copies of their own book as a marketing exercise?

I think there are less problems with my proposed system than some are making out. We're just asking for people of reputable standing in the publishing biz to make the same assessments publishing professionals make all the time about a book, with the exception that they're not being asked to invest in it.
 

PaulineMRoss

Inkling
So here I'd be thinking golden quills(?) Say your book gets a silver for a thousand sales. A gold for ten thousand. Platinum for a hundred thousand. Amazon processes the sales and when a book achieves the requisite number the quill is automatically stamped on the book page. And then readers see this, and automatically know that this book has sold a certain number of sales.

I quite like that idea. Any book that's sold a thousand copies (which have not been returned in disgust) is going to be at least readable, even if it's not necessarily the best thing ever written.

Some problems though. Firstly, the catch-22 - for an unknown author, how do you get sales when you don't have the quill? And how do you get the quill when you don't have the sales? So it doesn't help anyone to get those initial sales.

Secondly, how do you count sales? Total worldwide (if you could even get those numbers)? Amazon.com? All Amazons added together? Bookseller numbers? Publisher numbers? Amazon doesn't even release sales numbers at the moment.

What I could see happening is Amazon setting up its own system along these lines, without giving away how many actual copies have been sold. They are the people most in need of some kind of quality stamp of approval, and this is a blindingly cheap and easy system to implement that gives potential purchasers some confidence.

Nice idea.
 
Hi,

Dark One, this is a very simple system that requires none of the things your system does. There's no need for reviewers. No need for professional bodies to give a stamp of approval. No need for authors to pay anything. No need for readers to know the standing of the body doing the review. The only thing that's needed as I see it is a counter field on a computer. Naturally it's not going to be nearly as effective in reviewing books as your system. But the point is that it provides some evidence of quality where currently there's almost nothing.

And there's nothing to stop publishers buying up the first thousan copies. But they're doing that now just to get their books on best seller lists - eg Nigella Lawson's soon to be ex Mr. Satchi!

Pauline, as I see it this would work much as golden records do. Thus artists have gold records for the one song in different countries. So I would assume a book would have a quill in Amazon.UK and another in Amazon.com. It would only be an Amazon system, but since they're the lion's share of the ebook market that's probably ok. And no - it does nothing for authors trying to get their first books noticed. But that's not really the purpose of a quality seal. It's to give some reassurance to readers.

But, as I say, if you set the initial quill (silver?) quite low, say a thousand books, it's not that hard for a book with a proper edit, good cover and decent blurb to reach it, whereas hopefully the others won't. Or not for fifty years!

Cheers, Greg.
 
Last edited:

Devor

Fiery Keeper of the Hat
Moderator
I think any credential that only tracks sales is going to favor traditionally published or established authors. Music companies use "Gold Albums" to spur sales for their biggest records. They don't use it to promote the little guy with talent. I don't oppose the idea, per say, but I don't think it does the job of sorting the slush pile. It only shows which ones have already kicked off the slush.

To put out other ideas, a company like Amazon could crowdsource the job, but you'd need a complicated way of ranking not just the books, but the reviewers. Like maybe, if a reviewer has a dozen reviews that each get a dozen-or-more likes, then that reviewer could check off books to receive a this-book-is-readable star. A book which gets two stars gets a label. That kind of thing.
 
Agree with the point about the quill system favouring established publishers. We're looking for a system that helps the talented little guy.

Although, do grudgingly agree that the quill system would be better than nothing.
 
Hi,

My thought is that it does help the talented little guy. As long as they write well, edit properly, do good covers and blurbs, it should be as useful for them as it is for the trade pubbed. What it should knock out are those who simply don't want to put in the effort or really don't have a book in them.

Cheers, Greg.
 

Graylorne

Archmage
I'm with Dark One's line of reasoning. I want a system that tells people my books are ok, so that I can reach those first 1000 sales.
Last year I put four books on Amazon. My beta readers and the reviews I have say they are fun reads. They are professionally edited, with professional covers, reasonably priced, and getting them out there cost me a great deal of money. Last year I sold 54 titles in total. How long must I wait to reach those fabulous 1000 copies (per title, I suppose)? Then at least I'd have the initial costs more or less covered.
So those quills are nice for people who already have some traction. But it's not what I need, and many others with me. I want a reputable sign that tells the world I'm at least not writing crap.
 

C Hollis

Troubadour
So, when I consider this stamp of approval, I see one difference between it, and making it beyond the slush pile; sales. When an editor or agent peruse a manuscript, alongside their basic opinion is the question of whether they think the book will sell. They could very well believe the story is good, but not accept it because they don't think they can sell it.

Subjectivity is a very, very large part of traditional publishing.

If some organization was formed to provide the readers with a stamp of quality, sales should never come into the equation. What do they care if it sells, they won't have any vested interest in it. Right?

Add this thought. If one organization pops up with a stamp of approval, others will follow. Readers will look at a new title and think, "Well, the Dragon Guild approved this one, so it must be decent." or, "Ugh, the Hairless Halfling Horde approved this one, I don't usually care for those books."

Which leads me to this question for the group:

As long as one organization remains consistent with what they consider quality, why do we need to remove subjectivity from Dark One's proposed badge?
 

Philip Overby

Staff
Article Team
I agree that the quill system would be good to show how many sales someone has made, but I also agree that it's not really helpful for writers who are just starting out or don't have the presence to get that many sales. Of course that could be something to strive for and wouldn't be easy to achieve, so it may push authors to promote more to get the quill mark. There are loads of authors who go undiscovered who are very talented, but just aren't finding an audience for whatever reason. That's sort of how things go in any industry though I suppose. I guess what others are mentioning is how could this system be beneficial to brand new writers. It's kind of the idea of "you have to have sales to get sales."

I think what Pauline suggested of having a sort of Rotten Tomatoes for books would be a good idea. The problem is, Rotten Tomatoes takes reviews from both professional and user ratings. There would have to be enough professional reviewers to read every single independent books that comes out and that's just not feasible as of now.

Going back to the original idea, what if there was a sort of independent/indie writers' association that performed these services? You pay a monthly fee and get loads of feedback. The association could put some kind of stamp so that if readers pick up the book they know it's been through the ringer, edited, and polished to the best of its ability.

I don't know, just throwing ideas out there still. A lot of great ideas are being thrown around and it's good to discuss something anyway.
 
Top