Romulus ratings

"Lem" wrote in message
...
At http://www.romulus2.com/feedback/chart.php?1 there is a series of
ratings of good UK suppliers.

What I don't understand is how some suppliers have got such a high
ranking but they have a mediocre score.

For example, Dabs gets a score of 4.66 and a ranking of 19 but that
4.66 is worse than several other suppliers which are lower down in
the rankings.

Can anyone explain this as I like to use this page a lot.

5 stages

1.Suppliers with a minimum of 15 votes (reviews) in past year
2.Suppliers with a minimum of 15 votes (reviews)
3.Suppliers with less than 15 votes (reviews)
4.Suppliers which have been archived (denoted by )
5.Suppliers with no votes (reviews) at all

So a supplier with a low rating but over 15 votes would be further up than
one with a higher rating but under 15 votes.

"BUFF" wrote:

For example, Dabs gets a score of 4.66 and a ranking of 19 but that
4.66 is worse than several other suppliers which are lower down in
the rankings.

Can anyone explain this as I like to use this page a lot.

So a supplier with a low rating but over 15 votes would be further up than
one with a higher rating but under 15 votes.

Does make the rankings somewhat misleading imho - basically a company
with 100 people agreeing they're sh*te would get a better ranking than
a competitor with 10 customers saying they're the best company they
ever dealt with.

I can see some of the logic behind it but as more reviews have been
added to the site the flaws in the basic ethos behind the ranking
system have become more apparent. I still use it occasionally to read
some of the reviews but completely disregard the rankings.

--
iv Paul iv

[ Mail: ]
[ WWW: http://www.hopwood.org.uk/ ]

"Lem" wrote in message
...
"BUFF" wrote:

"Lem" wrote in message
...
At http://www.romulus2.com/feedback/chart.php?1 there is a
series of ratings of good UK suppliers.

What I don't understand is how some suppliers have got such a
high ranking but they have a mediocre score.

For example, Dabs gets a score of 4.66 and a ranking of 19 but
that 4.66 is worse than several other suppliers which are lower
down in the rankings.

Can anyone explain this as I like to use this page a lot.

5 stages

1.Suppliers with a minimum of 15 votes (reviews) in past year
2.Suppliers with a minimum of 15 votes (reviews)
3.Suppliers with less than 15 votes (reviews)
4.Suppliers which have been archived (denoted by )
5.Suppliers with no votes (reviews) at all

So a supplier with a low rating but over 15 votes would be
further up than one with a higher rating but under 15 votes.


Strange thing is that the first 34 companies on

http://www.romulus2.com/feedback/chart.php?1

have over 15 votes. And yet low scoring comanies are still ranked
above high scoring ones.

Something doesn't seem right to me.

See what you mean now - it seems to work OK down to no. 24 (i.e. lower
score= lower place) but then goes wonky for the next 10 whwere some should
be higher up.
Why don't you drop him a line &
1) point it out - he may not have noticed
2) ask for an explanation

Strange thing is that the first 34 companies on

http://www.romulus2.com/feedback/chart.php?1

have over 15 votes. And yet low scoring comanies are still ranked
above high scoring ones.

Something doesn't seem right to me.

See what you mean now - it seems to work OK down to no. 24 (i.e. lower
score= lower place) but then goes wonky for the next 10 whwere some
should
be higher up.
Why don't you drop him a line &
1) point it out - he may not have noticed
2) ask for an explanation


I have noticed ;-)

It may *look* wonky but actually it's doing exactly what it's supposed to
do. It's following the 5-stage rule noted by you (Buff) in your previous
post:

1.Suppliers with a minimum of 15 reviews in past year
2.Suppliers with a minimum of 15 reviews
3.Suppliers with less than 15 reviews
4.Suppliers which have been archived
5.Suppliers with no reviews at all

The top 24 on the chart have had 15 or more reviews in THE PAST YEAR. The
next 10 (the ones that look wonky) have had 15 or more reviews BUT (and
here's the crunch!) not in the past year. The firms after that (from 35
down) have less than 15 (the minimum) reviews or no reviews at all.

I felt that for the chart to be meaningful and up-to-date each company must
have at least 15 reviews to be at the top of the chart (otherwise only two
or three reviews - perhaps even posted by friends of the company owner -
could put a two-bit firm in number one position! That actually happened on
the old Feedback Arena before I rewrote the script.).

I also felt that the rankings must reflect the CURRENT performance of a
company. For instance, a firm which may have performed very well over a year
ago may have had 15 or more positive reviews at that time. Their performance
may have dropped abysmally since then (I can think of a few companies whose
service has declined in the past year) but unless 15 or more people have
posted reviews more recently, they would still be ranked highly.

It also works in the opposite direction of course. Some companies may be
performing much better now than they were over a year ago. Thus I introduced
the rule where the 15 or more reviews must have been posted in the past
twelve months for the company to reach the highest rankings.

Also, the ranking achieved by reviews over the past year supersedes earlier
reviews - so the ranking of a company is constantly updated as time goes on.
See: http://www.romulus2.com/feedback/fan...57968547,61675,

I know it's complex and it ties my head up in knots too sometimes! But it
does produce the fairest and most meaningful results. The code for a simple
highest-rating-gets-highest rank chart would have been waaaayyy easier to
write but utterly meaningless as it would not reflect the NUMBER of reviews
posted or how RECENTLY they were posted.

I hope that helps ...


David Knell

Feedback Arena
http://www.feedbackarena.com







"BUFF" wrote in message
...

"Lem" wrote in message
...
"BUFF" wrote:

"Lem" wrote in message
...
At http://www.romulus2.com/feedback/chart.php?1 there is a
series of ratings of good UK suppliers.

What I don't understand is how some suppliers have got such a
high ranking but they have a mediocre score.

For example, Dabs gets a score of 4.66 and a ranking of 19 but
that 4.66 is worse than several other suppliers which are lower
down in the rankings.

Can anyone explain this as I like to use this page a lot.

5 stages

1.Suppliers with a minimum of 15 votes (reviews) in past year
2.Suppliers with a minimum of 15 votes (reviews)
3.Suppliers with less than 15 votes (reviews)
4.Suppliers which have been archived (denoted by )
5.Suppliers with no votes (reviews) at all

So a supplier with a low rating but over 15 votes would be
further up than one with a higher rating but under 15 votes.


Strange thing is that the first 34 companies on

http://www.romulus2.com/feedback/chart.php?1

have over 15 votes. And yet low scoring comanies are still ranked
above high scoring ones.

Something doesn't seem right to me.

See what you mean now - it seems to work OK down to no. 24 (i.e. lower
score= lower place) but then goes wonky for the next 10 whwere some
should
be higher up.
Why don't you drop him a line &
1) point it out - he may not have noticed
2) ask for an explanation

Does make the rankings somewhat misleading imho - basically a company
with 100 people agreeing they're sh*te would get a better ranking than
a competitor with 10 customers saying they're the best company they
ever dealt with.

Paul, I hope my reply to Buff answers your misgivings, particularly my
paragraph:

"I felt that for the chart to be meaningful and up-to-date each company must
have at least 15 reviews to be at the top of the chart (otherwise only two
or three reviews - perhaps even posted by friends of the company owner -
could put a two-bit firm in number one position! That actually happened on
the old Feedback Arena before I rewrote the script.)."

As I'm sure you'll agree, any survey can only be meaningful if a reasonable
number of people are polled. Think of the ranking of a company as a survey.
Would you place any trust in a survey based on less than 15 people?

David


"Paul Hopwood" wrote in message
...
"BUFF" wrote:

For example, Dabs gets a score of 4.66 and a ranking of 19 but that
4.66 is worse than several other suppliers which are lower down in
the rankings.

Can anyone explain this as I like to use this page a lot.

So a supplier with a low rating but over 15 votes would be further up
than
one with a higher rating but under 15 votes.

Does make the rankings somewhat misleading imho - basically a company
with 100 people agreeing they're sh*te would get a better ranking than
a competitor with 10 customers saying they're the best company they
ever dealt with.

I can see some of the logic behind it but as more reviews have been
added to the site the flaws in the basic ethos behind the ranking
system have become more apparent. I still use it occasionally to read
some of the reviews but completely disregard the rankings.

--
iv Paul iv

[ Mail: ]
[ WWW: http://www.hopwood.org.uk/ ]

"RomQ" wrote:

Does make the rankings somewhat misleading imho - basically a company
with 100 people agreeing they're sh*te would get a better ranking than
a competitor with 10 customers saying they're the best company they
ever dealt with.

As I'm sure you'll agree, any survey can only be meaningful if a reasonable
number of people are polled. Think of the ranking of a company as a survey.
Would you place any trust in a survey based on less than 15 people?

My statement above still applies. It's true you need a decent number
of contributions for a ranking to be meaningful but it's nonsensical
imho that a company with a huge number of negative responses can be
ranked more highly than one who has predominantly good ratings from
fewer people.

For example you have Time, Watford and PC World with an abysmal
ratings contributed by a large number of people (thus they can
probably regarded as being crap companies to deal with) being better
rated (and climbing!) higher than for example, Evesham, Insight and
Maplin, each of which has an average score in excess of DOUBLE the
former companies. Then you have companies with a dozen or so
customers giving them glowing reports, resulting in average scores of
8 or higher, languishing outside the top 100.

A ranking system can only really be regarded effective if the end
result is a chart with the most favourable at one end of the scale and
least at the opposite end, and those in between being sorted in
(approximate) order. Ignoring the ethos behind the ranking system for
a moment, can you really say the Romulus really gives a true
representation of the suppliers listed, or even a reliable Top 100?
Companies with infamously terrible service like Time, Watford and PC
World are in your top 10%, with terrible reviews and an overall rating
which reflect the true nature of those companies, are ranked above
well-regarded companies with significantly better scores such as
Evesham, Overclock.co.uk, Extreme Cooling, Digi-UK, MicroWarehouse
etc.

As I said, I find the site useful for the overall rating and to read
the reviews people have posted, but the ranking system is so wide of
the true picture that I consider it to be of no practical use. When
it comes down to it it's your site and if the ranking system orders
them how *you* consider each of the suppliers then it's probably
successful, I just happen to disagree with your evaluation. ;-)

--
iv Paul iv

[ Mail: ]
[ WWW: http://www.hopwood.org.uk/ ]

When
it comes down to it it's your site and if the ranking system orders
them how *you* consider each of the suppliers then it's probably
successful, I just happen to disagree with your evaluation. ;-)

You're being a little unfair, Paul. You know very well that Feedback Arena
has absolutely nothing to do with *my* evaluation or how *I* consider the
suppliers. I have no control over the rankings whatsoever. I merely choose
the minimum number of reviews that I feel represents an adequate poll for a
rank to be reliable. The number could be 3 or 5 or 10 or whatever but I
think 15 is about right. I could be wrong and 10 has been suggested as
better - but I don't want some tiny operation to just get ten friends to
post great reviews for them and shoot them to the top. The larger the
number, the less that will happen.

Incidentally, the American ResellerRatings.com uses a minimum of 30 ("The
resellers must have at least five reviews for every month in the past six
months to qualify for the top seller list")
http://www.resellerratings.com/faq.pl

I would like to see companies like Evesham, etc. properly ranked in the
chart too but until enough people post reviews for those companies within
the past year for the results to be statistically reliable, there's nothing
I can do about it. As I said, *I* don't control the rankings, the site users
do. The more people who use the site and post reviews, the better it gets.

A ranking system can only really be regarded effective if the end
result is a chart with the most favourable at one end of the scale and
least at the opposite end, and those in between being sorted in
(approximate) order.

Regardless of how many reviews they get or how long ago they were posted? A
ranking system where a company with only two reviews posted by mates of the
owner two years ago could be ranked way higher than one with 50 reviews
posted by bona fide customers in the last month? That's not the kind of
ranking system I would regard as effective or want to place any confidence
in. ;-)

Regards,
David


"Paul Hopwood" wrote in message
...
"RomQ" wrote:

Does make the rankings somewhat misleading imho - basically a company
with 100 people agreeing they're sh*te would get a better ranking than
a competitor with 10 customers saying they're the best company they
ever dealt with.

As I'm sure you'll agree, any survey can only be meaningful if a
reasonable
number of people are polled. Think of the ranking of a company as a
survey.
Would you place any trust in a survey based on less than 15 people?

My statement above still applies. It's true you need a decent number
of contributions for a ranking to be meaningful but it's nonsensical
imho that a company with a huge number of negative responses can be
ranked more highly than one who has predominantly good ratings from
fewer people.

For example you have Time, Watford and PC World with an abysmal
ratings contributed by a large number of people (thus they can
probably regarded as being crap companies to deal with) being better
rated (and climbing!) higher than for example, Evesham, Insight and
Maplin, each of which has an average score in excess of DOUBLE the
former companies. Then you have companies with a dozen or so
customers giving them glowing reports, resulting in average scores of
8 or higher, languishing outside the top 100.

A ranking system can only really be regarded effective if the end
result is a chart with the most favourable at one end of the scale and
least at the opposite end, and those in between being sorted in
(approximate) order. Ignoring the ethos behind the ranking system for
a moment, can you really say the Romulus really gives a true
representation of the suppliers listed, or even a reliable Top 100?
Companies with infamously terrible service like Time, Watford and PC
World are in your top 10%, with terrible reviews and an overall rating
which reflect the true nature of those companies, are ranked above
well-regarded companies with significantly better scores such as
Evesham, Overclock.co.uk, Extreme Cooling, Digi-UK, MicroWarehouse
etc.

As I said, I find the site useful for the overall rating and to read
the reviews people have posted, but the ranking system is so wide of
the true picture that I consider it to be of no practical use. When
it comes down to it it's your site and if the ranking system orders
them how *you* consider each of the suppliers then it's probably
successful, I just happen to disagree with your evaluation. ;-)

--
iv Paul iv

[ Mail: ]
[ WWW: http://www.hopwood.org.uk/ ]

"RomQ" wrote:

When
it comes down to it it's your site and if the ranking system orders
them how *you* consider each of the suppliers then it's probably
successful, I just happen to disagree with your evaluation. ;-)

You're being a little unfair, Paul. You know very well that Feedback Arena
has absolutely nothing to do with *my* evaluation or how *I* consider the
suppliers. I have no control over the rankings whatsoever.

I wasn't implying you have any influence over the feedback itself, but
you *do* write the coding behind the ranking mechanism and based on a
set of rules you determine so the rankings are, in effect, of your
making.

I merely choose
the minimum number of reviews that I feel represents an adequate poll for a
rank to be reliable. The number could be 3 or 5 or 10 or whatever but I
think 15 is about right. I could be wrong and 10 has been suggested as
better - but I don't want some tiny operation to just get ten friends to
post great reviews for them and shoot them to the top. The larger the
number, the less that will happen.

The most effective might be some kind of weighted average, which used
a "balanced scorecard" approach to give each supplier a scoring, which
in turn was used to rank them. Each review would be a given a degree
of "trust", possibly based on the number of reviews posted by the
reviewer (avoiding a supplier getting 10 friends to post positive
remarks), age and quantity of reviews. I've also seen other sites
make very effective use review feedback, reviews of reviews as it
were, so that other visitors can state their level of agreement in a
review without necessarily placing a review themselves.

Another simpler alternative would be to simply leave suppliers you
consider not to have sufficient feedback as "unranked", rather than
try to rank suppliers using incomplete data.


A ranking system can only really be regarded effective if the end
result is a chart with the most favourable at one end of the scale and
least at the opposite end, and those in between being sorted in
(approximate) order.

Regardless of how many reviews they get or how long ago they were posted? A
ranking system where a company with only two reviews posted by mates of the
owner two years ago could be ranked way higher than one with 50 reviews
posted by bona fide customers in the last month? That's not the kind of
ranking system I would regard as effective or want to place any confidence
in. ;-)

As it stands the current ranking system isn't one I'd suggest you
could regard as effective or want to place any confidence in. Unless
you really think we should be discouraging people from doing business
with, for example, Evesham and suggesting everyone goes out and buy
from Tiny instead? ;-)

The site is very useful for gauging general service level of a
customer using the reviews and overall rating but, imho, the chart is
quite clearly NOT a true reflection of the comparative merits of the
suppliers listed.

--
iv Paul iv

[ Mail: ]
[ WWW: http://www.hopwood.org.uk/ ]