Advanced Map Appraisal System

[Tobius]

Hello to everyone who loves playing maps from the internet!

With this thread I'll introduce the Advanced Map Appraisal System (AMAS) which in my eyes is the ideal tool to appraise maps in a transparent and highly objective way. Here you can ask questions, give fair comments, and suggestions for improvement of the system. At first I'll illustrate the context which lead to the development of this system. Next I'll explain the basics that are important for understanding, then I'll take a look on single-player and later on multiplayer maps, and last but not least I'll exemplify the whole thing with a map.


The Context:
Many a person, who downloaded any user made maps from the online archives, surely have thought about the quality of the just played map. Then you're reflecting, but you simply come to a very subjective opinion which even affects your comments and rating of the map. But this is all wrong, because your opinion is not comparable to any other opinion or rating. Next you give the map a rating from 1-10, which finally is nothing at all if it is not explained in more detail with a comment, because these numbers say nothing about the strengths and weaknesses of the map. You are unable to make head or tail of it...

This situation brought me to the verge of despair time and again, because without any detailed comments and when I say 'detailed' then I mean really long texts I cannot make my opinion understandable anyway. But even these detailed reflections don't avouch a objective rating how I had to find out myself, because a lot of ratings I gave I wouldn't give them again today. Why do I speak about the past? Well, because I found a solution for this problem: the Advanced Map Appraisal System.


The Basics:
This Appraisal System isn't quiet simple, because for a objective and transparent map appraise a certain grade of complexity is needed since Heroes maps that you want to appraise are complex structures, too, which great complexity cannot be described with a lonely number.

The Advanced Map Appraisal System tries to describe a map holistic way, i.e. to slip all the different spheres of a map in this new system. Henceforth I'll call these 'spheres' simply categories. To ensure not to compare apples and oranges I differentiate two kinds of categories: main-categories and sub-categories. The sub-categories are immanent parts of the main-categories and make it possible to describe the main-categories in a sufficient and global way. At this point I want to add an example for making this partition better understandable:

We reckon that the quality of an apple shall be appraised. Therefore you need to define what makes out the quality this apple. I think that you can agree that different things define the quality. OK, let’s say that the quality is defined by the form, the colour and the taste of the apple, then these would be the main-categories of the Advanced Map Appraisal System. Moreover you should agree that the apple’s form, colour and taste themselves are defined by other things, e.g. diameter (form), colour saturation (colour), and sourness (taste). In a conclusion by analogy these terms are the sub-categories of the Advanced Map Appraisal System.

I think that this example clearly shows that you can decide a holistic verdict only in a cooperation of the different categories which define the map’s overall quality. In addition the differentiating of the categories leads to much more transparency what you cannot say about a abstract number which you find the archives today.

Concerning Heroes maps there is another difference that generally relates to all maps. There are two fundamentally different kinds of maps: the ones that are made for single-player experience and the others made for multiplayer games what from we differentiate into single- and multiplayer scenarios. In virtue of the different recipients both kinds of maps follow totally different basic approaches to satisfy the interests of their respective addressees. Hence you can deduce from it that a multiplayer scenario won’t be as much fun for a single-player as it could be if it was played by several human players what analogically applies to single-player scenarios. As a consequence you need to find different categories for single- and multiplayer scenarios to appraise them both in a holistic way. But this doesn’t mean that you couldn’t never ever appraise a multiplayer scenario’s quality by using the categories of a single-player scenario. The obvious question is how meaningful this would be under pragmatic aspects. The question arises where your own focus lies, i.e. whether you are more likely to play single- or multiplayer maps. However, here is the special case that you must not use the multiplayer scenario’s categories for a single-player scenario, because the mechanics of the game are simply not allowing this try.


The Advanced Map Appraisal System – Single-Player:
After the basics of the new system should be known with all readers, I can do the next step and explain how these theoretical thoughts are reflected within the Advanced Map Appraisal System.

At first I have to say that the evaluation unit of each category reaches from 0-100 points which can be considered as percentage as well so that 0 points are the minimum and 100 points the maximum of gainable points. There is a basically great differentiation possible due to the potential of 100 points in each category. This differentiation is impossible within today’s simple rating systems. Furthermore this distribution of points gives a much greater transparency, because with one look you can detect differences which were simply impossible to outline before.

As you read in the basics there is the need of different main-categories for a holistic appraise of maps. I’ve chosen a triumvirate: Graphics, Map design, Atmosphere. In the following I’ll examine each of the main-categories and their associated sub-categories.


1. Graphics
The Graphics describe the aesthetics of a map. It predicates how well the map has been designed. There should be quantitative and qualitative considerations. The actual score of this main-category determines the overall score to 30 %. For valid data we use the following sub-categories:

1.1 Details:
This sub-category makes quantitative considerations concerning the richness of detail of the map which please the eye. The amount of map objects is decisive, i.e. number of trees, mountains, meadows, etc. simply everything that increases the beautification of the map.

The score depends on the amount of map objectives. At this point it is not important whether it is all the same objectives (this is left for the next sub-category), but the amount is highly decisive. But what is ‘richness of detail'? Well, the other thing was a map with stark and empty areas. However, such a map can be diversified anyhow by giving each square a different terrain, e.g. grass, lava, snow, etc.

The actual score of this sub-category determines the overall graphics-score to 40%.

1.2 Diversity:
This sub-category describes how diversified the map is designed. For valid data we need another quantitative analysis for ascertain how many different map objectives are placed on the map. Herein you must not forget the terrain, because these have to be varied, too, otherwise a high diversity seems to be almost impossible.

There is a clear difference between Diversity and Detail. The evidence for this is the fact that you can design a map full of detail but without any diversity. How could this be? Well, imagine a landscape that solely consists of ‘grass’ terrain (wherewith I mean the textures of the ground). In addition this map was strewed all over with the same type of bushes. Due to the numerous bushes there was a high detail, but it was the same bush all the time. That is why such a map was completely poor concerning diversity.

The actual score of this sub-category determines the overall graphic-score to 40%.

1.3 Overall landscape:
This is the last sub-category of the first main-category ‘Graphics’. At this point we consider the asymmetric or symmetric design of the map. Since your eyes get tired of a symmetric map whose landscapes moreover aren’t looking really natural, a map gets the higher score the more asymmetric it is designed, i.e. ‘natural’ landscapes are all the better. What counts is a ‘handcrafted’ design no ‘obstacle tools’. Maps that aren’t made completely by their authors can’t reach really high score, because they simply lack naturalness.

Furthermore there are other criteria aside from asymmetry and symmetry which are important at this point. Apart from geometric forms (which are unnatural in contrast to flowing forms which underline the naturalness) the utilize of available space can be important in some cases, e.g. a huge map with a little isle on the surface and one dungeon in the underground is simply bad designed whereas a map whose surface and underground are entirely utilised is much better.

Moreover counts the harmony of the placed objects and terrains. Does a map appeal overloaded, because objects seem to be placed by hazard it can destroy the aesthetics of the map. Such deficit would reduce the score since harmony is something natural.

The actual score of this sub-category determines the overall graphics-score to 20%.


2. Map design (Concept):
This is the second main-category of the Advanced Map Appraisal System. I’ve heard from other people that the term ‘Map design’ could be irritating so I added ‘concept’ to clear things up. I hope you can follow my thoughts and tell me which term is the better. In this case I watch for your comments.

This category shall describe the way a map is designed (not graphically!). With ‘designed’ I mean the way its concept is realised. Here we examine whether the map is thought-out, coherent, challenging, and innovative. The actual score of this category determines the overall score to 40% thus it is the most important category of the three main categories. The reason for that is the significant question whether the map sinks into memory or not. If you get involved with the map, than rather because of its complexity and play depth than its commonness, and such a map sticks to your thoughts thus this is the main-category with the highest weighting. For valid data we have to differentiate the following sub-categories:

2.1 Complexity/Play depth:
This sub-category examines how thought-out the map is whereby the premise is that more complex maps with a higher depth are of greater quality than maps that are just about slaughtering your enemies (those maps I call at times “Slaughter maps”) thus the play depth is quiet little.

The score is determined by various factors. One of these factors is the map’s structure. This begs the question if the map is quiet simple to ‘read’ thus there are no challenges that need any thinking to proceed. Maps with puzzles have big advantages concerning that criteria, but there are other factors, e.g. quests. Are there any tasks at all and if yes in which quality? Are these quests fascinating and do they enrich the map? Here it is important to look for quality and quantity, because both increase the play depth as well as the complexity of the map. However, quests and their texts can give the map a much higher depth so you must not forget that point.

The actual score of this sub-category determines the overall map design-score to 60%.

2.2 Challenge:
This sub-category gives a assessment of the map’s degree of difficulty. This is probably the most subjective category, because every Heroes player is more or less experienced with playing the game. This is the reason why this category is not so high weighted like other ones. In spite of this it’s indeed a important category, because nobody likes playing maps that are much too easy to win without any challenges. I want to add that good as well as not so good players have to think outside one’s own box when they give points to ensure a objective appraisal as possible. For this category the map gets more points the higher the difficulty is.

The actual score of this sub-category determines the overall map design-score to 20%.

2.3 Innovations:
This is the last sub-category of the main-category ‘map design’, but that does not mean this category was unimportant not at all. Where lies the importance of this category? Well, you don’t want to play maps where you’ve seen everything already and maybe even better. With ‘Innovations’ I mean, e.g. a new ‘own’ story, exceptional quests, new situations at all, etc. A special case are maps that are released as a sequel to an older one. With such maps you have to consider the alterations compared to its predecessor. You have to make sure that it’s not simply a boring rehash. For this category the map gets more points the more innovations you can find.

The actual score of this sub-category determines the overall map design-score to 20%.


3. Atmosphere:
This is the last but not least main-category of the Advanced Map Appraisal System. This category considers the quality of the atmosphere the map (tries to) create, to rivet the player, and animating him to play on and on. Thus it is easy to detect the importance of this category, though there are only two sub-categories, but these are of high percentage what underlines their significant role for a good map. The actual score of this category determines the overall rating to 30% and fulfils the trinity of the Advanced Map Appraisal System so all categories for the appraisal are detected now. The following sub-categories are determining the atmosphere-score:

3.1 Suspense/Surprises:
This sub-category answers the question to what extend the map motivates the player to proceed with playing thus it is one of the most important categories at all, because what is a map worth that is not played till the end? Not so much and mostly the reason is that the map simply lacks suspense, which could motivate you to play the map through.

This desire for playing the map is the touchstone for the score which the map reaches for this category. That can depend, for example, on the interactions between the players, because a map were you do a lot of walking, but nothing happens, isn’t quiet enjoyable, and so you don’t have that ‘desire’ to proceed with the map what correlates to a lower score at this point. You can find this problem of lacking suspense more often in linear maps than in non linear ones. Why is that? Well, strictly linear maps often loose their high suspense from the beginning with proceeding from month to month, so it finally comes to nothing. By the way this is one reason why it is not that easy to design linear map that keep their suspense till the end. At this point non linear maps have a fundamental advantage , because the interactions between the players are part of the principles of such maps.

The actual score of this sub-category determines the overall atmosphere-score to 60%.

3.2 Story/Dialogues:
This is the second and last sub-category that answers the questions concerning the quality of the atmosphere. Here the Story plays the crucial role. We want to examine with which quality the story is told. The score depends on the complexity and quality of the story.

For a high score it is important that the story is told with much of suspension and it has to be plausible. Are there any twists and turns, or surprises? A special case are humorous stories, because it’s much more subjective, because you have to share the author’s kind of humour to like the map. But this is not a problem of the Advanced Map Appraisal System, it’s simply a specialty the authors of these maps have to cope with. Furthermore there should be dialogues and direct speech, because you want to experience the story, and without any dialogues the map can’t get alive. So the dialogues must not be forgotten. Of course, maps that want to reach a really high score concerning the story have to present events, thrilling and entertaining quests as well as a good background, and the story has to correlate with the acting of the player. Maps that include episodic sequences, but no golden thread loose some points as well as maps whose story doesn’t correlate with the map itself.

However, I have to admit that there are regrettable few maps with a story, so you cannot weight this point as high as some others would like to do. For that reason the actual score of this subcategory determines the atmosphere-score to 40%.


Practical Use:
After the theoretical fundament of the Advanced Map Appraisal System is presented, I want to put it into practice. I am sure you were asking yourselves how the whole thing would finally appeal, but I assure it’s not half as complicated as it seems at a first glance and you really get convincing results out of it as it simply wasn’t possible before.

Before there’ll be any arguing I want to clear out some possible starting problems with the AMAS:

1. You need some experience with using the AMAS, because it’s reference based, i.e. the scores need to be actualised if there’s a map that sets new touchstones in some categories. After a while you get a feeling for it, and this teething trouble will disappear for good.

2. I don’t say the AMAS was absolutely objective, but in use of a experienced player you get the best objective results possible at all. No other system that I know is a match for the AMAS.

3. My advise to you is to write all your appraisals down (text-file, excel-file, etc.), so you can look it up if there should be any doubtful feelings concerning the scores. This allows you to make up a ranking which gives you the needed overview.

The logical premise for a convincing result is the fact that you’ve played through the map. You don’t need to play the map in another way than you did before, because you’ll recognize all particularities that are important for the AMAS while playing the map. Of course, you can look some things up in the editor after playing, but it’s no must, though you can make out nuances for what the AMAS is simply perfect.

If you know the other rating systems (if they deserve that name) then you probably see a problem concerning the comparability. But the difference between the overhauled 10-point-ratings and the AMAS isn’t as big as it seems to be at first sight. To clear things up I want to give you a little overview about the possible results of the AMAS and their weight:

Rating > 90%
It’s safe to assume these maps as real masterpieces, because they define new quality standards for their kind. These maps have proved a recipe for great fun that won’t release you not so fast. Every single hour playing is more than worthwhile. All you have to do is to download and play it!
Rating > 80%
These are maps of very good quality that are highly recommendable for those who appreciate quality: Graphics, Map design, and Atmosphere satisfy your claims, the overall package is consistent.
Rating > 70%
These are the maps of the second class: good quality, but there are better alternatives. If the various shortcomings had been eliminated, a higher rating and a recommendation was possible.
Rating > 60%
Here was ‘well-intentioned’ confound with ‘well-made’: either disappoint the Graphics or the Map design shows deficits. Also the Atmosphere seems to be a word of foreign origin here. You’re not missing much!
Rating > 50%
These maps are only just adequate: there are conspicuous deficits in one or more categories if not a boring game-play will frustrate you.
Rating< 50%
Well, where the great maps are worthwhile every hour of playing you will bitterly regret every single one. There are deficits across-the-board: you’re simply waste your time here!

Well, I think it’s time now to illustrate the whole system with an example. I’ve chosen the Heroes III map “Gold Heart” by Timothy Duncan. This map should be well known and you can download it almost everywhere. You well see, the AMAS is not that complicated as you might be reasonably expected:

Gold Heart – Advanced Map Appraisal System (single-player):

Graphics..........................83 (30% =24,9)
> Details...........................85 (40% = 34,0)
> Diversity........................77 (40% = 30,8)
> Overall-landscape...........93 (20% = 18,6)

Map design......................89 (40% = 35,6)
> Complexity/Depth...........92 (60% = 55,2)
> Challenge.......................71 (20% = 14,2)
> Innovations...................100 (20% = 20,0)

Atmosphere......................90 (30% = 27,0)
> Suspense/Surprises.........92 (60% = 55,2)
> Story/Dialogues...............86 (40% = 34,4)

Rating: 88%

As you can see “Gold Heart” is a great single-player map, because it has no serious deficits. The only thing is not so good Graphic-score, but it’s still very good. The final rating is simply the sum of the scores of the main-categories Graphics, Map design, and Atmosphere.


Closing Remark:
After reading all the information about the Advanced Map Appraisal System I hope you enjoyed reading it at least a little bit. I know for some of you it might be too much text, those should read it little by little and not at a stretch. However, I hope that you’ve got a comprehensive view about the AMAS which is in my humble opinion the best way to appraise maps as objective and holistic as possible. No 10-point-scale can match it, because what do get to know about the map from that 10 points? Almost nothing! The AMAS is completely different: you can see everything from importance with one view on the scores that is concerning transparency really unsurpassable, isn’t it?

Thank you very much for reading all the words I’ve written. Maybe you start thinking about map appraisal, too, and want to give some fair comments. I would appreciate it.

Greetings
Tobius

[Koni]

I had thought, that your detailed system would find a little bit more attention with at least a few comments. Where are you, players and mapmakers?

[Akul]

It takes some time for some people to read posts of this size.

[rdeford]

@ Tobius -- based on a very quick reading, this posting is just a quick reply to let you know your work is appreciated. This is a good, well thought out concept. More comments will follow as I get the time to peruse it in depth, and see what others have to say.

First comment is that I think "Map Concept" is greatly superior to "Map Design" for the purposes of an evaluation system. For many, many people the word design means artistic design, for most others it means the arrangement scheme of the map. In either case, folks won't be thinking of the concept for the map.

Next, I think "Originality" along with or instead of "Innovation" or perhaps as a subcategory under Map Concept needs to be a factor, a big factor, in the evaluation.

@ Akul -- yes, it does take time to read a long posting, plus, many of us don't visit the forums daily.

[Pitsu]

On what I see a sad and cruel reality:

Ordinary users are not even likely to come back after completing a downloaded map and rate it in 10 point scale. If you ask them to grade 9 different aspects of a map, they will just ignore you.

More complex and systematic map judgement criterias are needed only for people who either judge a mapmaking contest or do map reviews on regular basis. But again, most of these people prefer to develop their own criterias and algorithms since they rank different aspects differently.


That being said, I know how fun it is to work out a new good method even if there is no high demand for it. So some quick comments about your system.

I am unsure about the impact of some aspects. For example what about graphic diversity on a small sized well themed map vs XL world map? Should the small map always get lower grade because of not being as diverse graphically? I think graphics is actually very tightly related to the atmosphere. Kind of story and graphics being two sides of the same "atmosphere" coin. If it fits with the story there is IMO no need for highly diverse graphic.

Bugs and replayability are IMO varaibles that should be included in a good review. I think your current system does not consider them.

[Angelspit]

Ordinary users are not even likely to come back after completing a downloaded map and rate it in 10 point scale. If you ask them to grade 9 different aspects of a map, they will just ignore you.

More complex and systematic map judgement criterias are needed only for people who either judge a mapmaking contest or do map reviews on regular basis. But again, most of these people prefer to develop their own criterias and algorithms since they rank different aspects differently.

I was halfway through the post and about to say something similar, but Pitsu phrased it better. Even a very well-known map such as Fury of the Elements got only 28 votes here (and 43 at M4H), so it's unlikely a more detailed and time-consuming review system would be more popular.

[wimfrits]

Looks like a good system in theory

Pitsu already noted down the rest of what I wanted to say at this point.

[arturchix]

Indeed, which reminds me I basically never rate any of the downloaded maps. If after completion of the map the game would trigger a pop-up window of my internet browser, asking me to immediately rate the map (and not in 9 different aspects), I'd do it. Otherwise don't bother.

Also for a player it's a lot easier to rate a map in a simple scale from 1 to 10.

[Humakt]

Looks like a good system in theory

Pitsu already noted down the rest of what I wanted to say at this point.

Indeed. Sometimes I wonder why I even bother publishing maps anymore.

[Akul]

Indeed. Sometimes I wonder why I even bother publishing maps anymore.

Because mapmaking is fun or/and because it is a good feeling to know that people are playing your maps?

[Angelspit]

Mapmaking is much like taking vitamins: you do it but you're not so sure if it's really worth it.

[Humakt]

Lets put things on perspective, it takes at least 1000 times effort to submit and publish map compared to simple click-rating system on 0-10 scale.

And same 1000:1 ratio is propably indicative to the effort of making the map compared to just playing the map.

Anyway, I like the rating system presented in initial post (even if it were pointless due to small activity on community's part general. though I feel the current rating system CH has is almost as equally pointless). It might need to be made simpler though.

Because mapmaking is fun or/and because it is a good feeling to know that people are playing your maps?

I agree on the first point, but only partially since map making is almost as hard as working and time consuming (and well, sometimes harder than working). I think Ururam Tururam said it best (trying to remember the phrase): "Map Making is 5% Inspiration and 95% perspiration". And the whole fun factor while making the map really comes from the inspiration. Well usually you can take pride of the finished areas if they're done well but that's pretty minor before the map is ready.

Of course then there is the factor of motivation. Without motivation, you won't go through the perspiration progress. There has to be something that motivates the map maker, or the worker. Whether it is a payroll, lash of whip on the back, inspiration, kind words, some greater purpose or goal, or some other reason.

I'm afraid that map maker (unless (s)he does it for the living) falls short on those motivation purposes. Inspiration does commit to the motivation of the mapper, as does the (headstrong) goalseeking (like making map to play it yourself or with your buddies). Yeah, it's pretty fast to make a bare-boned map that can be played quickly with your buddies. But when it comes to publishing maps, well, published map do require a lot more work if you hope to have any feedback from them at all (people doesn't seem to bother sending negative feedback hardly at all if doing it is even little hard for them. so the map has to be good in order to get even some feedback). Unfortunately feedback of any sort is minimal at best in HoMM community. Not that I have given feedback myself either as what little I play of HoMM is either playtesting my own map (and usually I do most of/solely the playtesting myself, as getting devoted testers is pretty hard) or some hotseat (which I play very rarely actually and frankly is also often played on my maps).

Second point you made is kind of moot: if you don't get feedback how you know people even play your map(s)?

Ah, I fear this rant seriously exceeded my montly forum message lenght/amount quota. Going to sleep now.

[arturchix]

Second point you made is kind of moot: if you don't get feedback how you know people even play your map(s)?

Count of downloads? I'm afraid a higher number of submitted ratings won't do any good as a feedback for the mapmaker.

[Vlaad II]

Hey, Humakt, what about the fame, fortune and scantily clad ladies that come with mapmaking?

Seriously though, I like the rating system at Age of Wonders Heaven:

http://aow2.heavengames.com/downloads/l ... =scenarios

Click on any file to see the grades and comments. So instead of getting e-mails, mapmakers would be able to get feedback on their respective map pages. Sometimes it's as short as "kewl map", but some fans actually take the time to thoroughly review your work. An "official" review (such as Psychobabble's or Qurqirish Dragon's reviews) could be placed on top of the page.

Rating maps on a 1-10 scale is often misleading; for example, I value aesthetics over balance - and how can I tell that from a 7.5 grade? In addition, there are people who seem to enjoy giving the lowest possible grades... The only scenario I'd give 1 is a completely empty, unplayable poor excuse for a map.

[Vlaad II]

Second point you made is kind of moot: if you don't get feedback how you know people even play your map(s)?

Count of downloads? I'm afraid a higher number of submitted ratings won't do any good as a feedback for the mapmaker.

For the record, I believe the download count at CH is bugged. That aside, I admit I like it when the number amounts to thousands. Then again - like ByteBandit noticed in another thread - there are certain patterns that are more successful than others. For example, casual singleplayers are fond of XL maps, while my M and S maps have created far less buzz (which is sad, since the size does NOT matter LOL).

Unfortunately, I cannot find Vel's topic about this - he started an excellent survey, and while I'm not sure whether he ever finished his project, he was definitely on to something...

[Humakt]

I'm afraid a higher number of submitted ratings won't do any good as a feedback for the mapmaker.

On current system, no, as I partly addressed it in the post above. But the comment system Vlaad told could be good.

And frankly, number of downloads might not really tell you whether the map has been success or not. It just might point out the map is xl (with underground even) or picture map as people download those most.

Hey, Humakt, what about the fame, fortune and scantily clad ladies that come with mapmaking?

Aw, my imagination can get only so far. You sir, have obviously out-imaginated me and I'm beaten. Here's your imaginative victory prize: