Policy Review Concept Submissions

HeaLnDeaL

Let's Keep Fighting
is an Artistis a Forum Moderator Alumnusis a CAP Contributor Alumnus
I don't have a ton of time to give a detailed response yet, but I'm hoping to flesh out my thoughts later. For now, I want to say I'm somewhat torn between the Blueprints and the Concept Improvement camps.

Overall, I'm very concerned that if people can see the "simulations" done in concept QCs that it will probably affect the actual CAP process and could potentially limit discussion rather than foster it. However, it then gets tricky on who should have access to viewing these, if not just the moderators (for the most part I think I can agree with srk's suggestion of making the QC forum to be hidden). I think a lot of the good CAP participants do some concept simulations of their own already, but making a public place to post and giving these otherwise private thoughts a visible avenue frankly worries me.

I'm not particularly worried that we won't be able to get a decent number of quality concept submissions, but moreso I'm a little skeptical of getting a group of QCers being able to agree on an appropriate number of "good" concepts. There would probably have to be a fairly explicit framework of what makes a concept good beforehand just to attempt to get people on the same page. Obviously we will all mostly agree that single-solution concepts are bad concepts, but there seems to be a number of differing opinions on what makes a concept good/bad. If this is to be overcome, having a set of agreed-upon criteria when QCing concepts is probably for the best...
 

paintseagull

pink wingull
is a Top Artistis a Forum Moderator Alumnus
Purely process thoughts here:
I agree that it seems like a whole subforum for this would take a lot of time and a lot of people that we probably don't have right now. Basically the point is that no one person should get to decide the slate and the slate should focus a lot less on what seems popular in the submission thread than in other parts of the process. I know that concepts asking the question "Is this Pokemon viable in OU?" have been slated in the past and this should NEVER happen.

Basically, we just need to be more strict about what constitutes a legal submission. And yeah, I could see us ending up with far fewer legal submissions and an even smaller slate in this case. But I think that would be a good thing. Right now there are way too many and it's very tough for a TL to pick a set.

Here's what I'd envision as a way to boil down the Blueprint/Concept Improvement: two concurrent threads. Concept Submissions, and Concept Workshop. In Submissions, if you post something that doesn't pass a very rigorous set of standards, your concept gets deleted and you don't get to post another. In Workshop, the standards that are enforced in the current ruleset for Submissions could be employed to weed out the complete garbage, while actual decent ideas could be thought through and improved by the community/veterans. In the end though, it's still the submitter's job to ultimately provide the finished concept that passes the Submission thread's standards. Then the TLT/mods could make a slate.

The above is still a lot of veteran/mod work but maybe less than the subforum suggestion, where it seems like we'd have to pick slates multiple times. One slate to work on, then a second slate to vote on.
 

Ununhexium

I closed my eyes and I slipped away...
is a Community Contributoris a Smogon Discord Contributoris a Contributor to Smogonis a Smogon Media Contributoris a Social Media Contributor Alumnusis a Forum Moderator Alumnus
What are the problems with current concepts that can be solved with dedicated attention in a subforum?
Personally, I don't think there is anything inherently wrong with the process as much as I find it hard to deal with. During the concept submission stage, I try to give feedback when I can and as much as possible, but I always find myself giving each concept too little attention or being too lazy (maybe "lazy isn't the best word") to keep going back and forth through the various pages to look at every single concept and try to give feedback, while the addition of a forum would make it easier to respond to a concept submission "one-on-one" and pay attention to that one at a time. It also just makes the whole stage much more organized and less messy. The reason I don't find a forum for other stages with submissions is because not as many people participate in those; meanwhile, many people post a concept for the given CAP.

What will be the process to improve concepts in the subforum and who will be involved?
I actually really like the idea of having a QC team to sort of pick through the various concepts in the forum. That said, I don't picture them as people who suggest changes then say "Do this then QC 2/3" or whatever, but more as just influential participants who are acknowledged by CAP leaders as people who will help make the concept more defined / more clear / etc. Basically, I don't envision this as a formal process as it is ultimately the TL who slates the concepts (though he or she should definitely ask the QC team for input), but rather as a collection of influential users who try to help out submitters.

What interaction or conflict will the subforum process have with the normal CAP construction process?
I think this will mostly have a positive effect on the process as a whole, because if we start with a solid base, we can better structure the process after it. This will definitely help determine the goals of the project early on so we won't have as many screwups later in the process, because concepts really do make or break the whole thing. Also, I think that the TL simply receiving input from the "QC Team" will definitely help in deciding what to slate, because one person isn't able to see all sides of it at once.
 

nyttyn

From Now On, We'll...
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
I want to say first and foremost that I agree there most certainly is not enough qualifying manpower in CAP right now to do everything we would want out of this thread. At the very best, we can probably get some people together to weed through concepts, pretty them up, make sure nothing bad gets through etc but that's it. We definitely do not have the manpower to do "CAP Simulations" to vet concepts.

Concept Discussion we MIGHT be able to do, but it's going to have to happen primarily on IRC. Let's face it - we don't really have the time to sort through all the submissions we normally get with a forum alone. Unless we take the potentially biased route and only discuss concepts that a member of the QC team brings up as having potential, at least the preliminary stages of weeding/selecting are going to take something where communication and feedback can happen rapidfire - IRC. After that, if we can get it down to <10 concepts we can probably discuss them on an individual basis using the forum.

Frankly TBH at this point I favor the "subforum, you get one thread/concept submission per CAP" solution, if only if we can easily determine what the bad concepts that aren't worth bringing up are are via nobody of importance replying to them.


ok i'm tired and it's late but i really wanted to throw my thoughts out here. if you take nothing else from this post just take the fact that we need to remember we're frankly working with a very, very limited pool of actually competitive players at this point, and manpoweer is a huge concern that needs to be brought into consideration for any future ideas/proposals (sorry DougJustDoug, your proposed "playing it forward" plan just isn't realistic with our current resources).
 

Birkal

We have the technology.
is a Top Artistis a Top CAP Contributoris a Top Smogon Media Contributoris a Site Content Manager Alumnusis a Battle Simulator Admin Alumnusis a Senior Staff Member Alumnusis a Community Contributor Alumnus
The argument that we "don't have enough manpower" is both limited in scope and disheartening to hear. The big talkers in #cap and the Topic Leadership Team hardly make up the majority of our contributors. Take for example every single Concept Submission stage: we get people coming out of the woodwork submitting great concepts and feedback. I can't stand by the logic that we shouldn't do something that would greatly improve CAP because we don't think there are enough participants. There will be.

And in terms of there not being enough QC accepted concepts, I fail to believe that as well. Many concepts could be easily copy / pasted from past concept submissions and brought to us. I'm actually more concerned that we'd have too many to look at! There are plenty of people out there who like to submit interesting concepts for us all to consider.


Now, I'm going to work under the assumption that we agree with Doug's overall proposal, particularly the "what makes a good concept" section. Let's try to fill in some gaps.

1) Concepts are submitted to a stickied OP. Anyone can do this. In order to make it into QC Phase 1, a QC teammate must create the OP with their concept. This means that concept submitters must convince at least one QC member that their concept is worth pursuing. In a way, this serves as a check itself.

2) Concept thread gets posted (just a copy / paste by a QC member). Anyone can discuss the concept's merits and pitfalls. They are approved for simulation just like C&C. Getting a QC approved 3/3 is what we're aiming for here. Note that this approval isn't the final stage. It's mostly an approval for, "this concept is worthwhile enough for us to spend time simulating it."

3a) Approved concepts are simulated by the QC team. I'm not a huge fan of creating a private forum to simulate each stage of the concept. Rather, this sounds like something that would be much more efficient as a speed CAP on IRC. QC members can have their own private chat for simulating any given concept. I think the last thing the QC team will want to do after posting in the Concept OP and in each concept thread would be to post in yet another thread. Just let 'em hash it out on IRC; it's quick and painless. Once completed on IRC, QC members can PM it to other members to see their agreement.

3b) If the concept is approved, the IRC log of the simulation is never released to the public. But, if it is rejected, the simulation(s) should be posted to explain why. I think the public would find it interesting to see why an approved concept didn't ultimately work. It would be a great learning experience for concept submitters.

4) At this point, the QC team notes any specific pitfalls the concept may have and marks it with a [DONE] tag, ready to be selected by the Topic Leader.​


Thoughts?
 

nyttyn

From Now On, We'll...
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
Birkal, can we please be realistic here? As many "ideas people" as we have, it's been acknowledged and established over and over we lack competitive people - and that's what we really need for this to be a realistic idea. If you have people in mind, go ahead and name them, and put the fears/concerns of a lack of manpower to rest. That's what has killed so many of our PRC ideas in the past - a lack of manpower, and so far you've provided us zero reason to believe this would be different, especially since we need a higher quality of user for this particular agenda (to the point where even some of the mod team wouldn't qualify!). And no, you can't just point to Concept Submission stages - not only do those get way more total users, many of them are one and done posters who never contribute to that particular CAP again. They might come out of the woodwork for that step but they sure as heck go back to the woodwork once it's over.

And as a total side note, the big talkers in #cap and the TLT also make up the majority of people who actually participate in multiple competitive stages of any given CAP. I don't think we should even consider anyone who doesn't participate in multiple stages of CAP (due to both a lack of experience in participating in CAP itself as well as a lack of dedication to the competitive side of the project), and the number of people who aren't hot in #cap/tlt and participate in multiple competitive stages is rather small. A number that gets only smaller if we only consider people who are competitively qualified (Meaning they actually play OU and are actually good at it/knowledgeable about the metagame).

Secondly, again, I must stress that we're not going to be able to do CAP simulations. This is less of a "do we have the manpower" question, and more "nobody is going to be arsed because it is way too much work" question, as well as a question of logistics. I've done flash CAPs before, hell, I've organized one. While fun, the bloody things take up entire afternoons. Ignoring flavor stages, you need to debate a typing, which requires going through the OU threatlist/usage stats and seeing what's hot, you need to debate an ability (and there are a lot of abilities), and you need to debate moves (and there are a HELL of a lot of moves). Even a streamlined version would likely take at least an hour, if not hours, of totally uninterrupted participation (and it'll probably take hours since people aren't going to be totally uninterrupted) - and as it stands, we're already having questions getting people to stay focused enough to participate in PRC.

Okay, numbers time. I'm going to entirely discount CAP 19 (which still had a crapload of concepts submitted) as it was a complete anomaly insofar as participation goes, and instead go to the prior CAP, CAP 18, which was a lot closer to how many submissions we usually get (See: CAP 4 and 5 from last gen). I'm also going to ignore illegal/really bad submissions that would be instantly deleted like this one. We had 86 legal submissions (and that's me just trying to count them, there were so many to sort through I'm sure I missed at least one or two). Assuming even half of those made it past the first QC check, we'd still have 43 submissions to deal with and provide 3/3s for (in reality 2/2s, because obviously the person who let it past the first QC check is more than likely going to vote to let it past the second, otherwise why would they have vouched for it in the first place?). Let's also assume that only half of those make it past that stage - now we're left with 21 concepts to deal with. Not too far off from how many actually viable concepts we tend to get.

We already have participation problems in PRC as it is (and PRC is incredibly flexible insofar as posting goes), now imagine having to organize/put together a speed CAP on IRC for each and every single one of those. Even ten, an incredibly small number for how many concepts we get and lowballing to the extreme, would be a nightmare - that's ten separate occasions that at least a few members of QC have to be online, have blocks of time dedicated solely to the speed CAP, not be interrupted during that time frame, and be willing to do so that particular day. Now imagine having to deal with upwards of twenty or higher concepts.

Then we have the question of momentum. The simulation stage, assuming twenty concepts to simulate, is going to take bare minimum two thirds of a month if we manage to somehow average one per day (a really fast pace, all things considered seeing as how these things take hour(s) of uninterrupted time to do) - and it took considerably less time than that for CAP 19 to be irreversibly screwed on momentum. Sure, we could fix this by branding the concept assessment stage as Pre-CAP, but it's already been deeply ingrained into people's minds that Concept Submission = CAP start!, and I don't think we should bet on being able to change that perception. This is a bit of a reach insofar as concerns go, but I think after CAP 19 we are better safe than sorry insofar as regards to anything that could potentially murder momentum.

Even if we ignore that, we still have the problem of getting people to sit through ten, twenty, maybe even more flash CAPs, depending on how many submissions we get. Love for CAP only goes so far, and it is a considerable amount of work to sit through even one speed CAP. People won't have to be there for every single one, yes, but considering we need at least three people to do a speed CAP (and frankly even that seems like a dangerously low number of different view points), and considering that people will have to participate in multiple ones (which will be dependent on the size of the QC team), I'm not sure how you can go so far as to call it 'quick and painless', much less even a viable proposal in the first place.
 

DetroitLolcat

Maize and Blue Badge Set 2014-2017
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
Note: I've been a bit behind on this discussion, so I'm trying to get all of my thoughts about this subject into one post. The next three paragraphs are about why I believe a concept subforum is a good idea. After that, I discuss what should go in that subforum and how we should move forward with Concept Submissions.

The overall goal of this thread is to raise the quality of the average concept submission and create more quality concepts. I'm not going to discuss what makes a quality concept (Doug touched on that, and I agree with his definition), but rather how we can vet quality concepts and weed out the poor ones before they make it to the thread - or worse - the slate. At the moment we have a system of throwing concepts at the wall and seeing what the TL thinks will stick. That phrasing might not be entirely fair to the current system (it has produced some good concepts in the past), but that's essentially what we're doing. The problem with that system is that it's extremely difficult for the TL - one person - to decide which concepts have and do not have potential (to the point where I'm convinced it's a crapshoot whether not the concept ends up "working" or not) and that the TL has essentially zero margin for error when it comes to slating concepts. One crappy concept on the slate can doom a project before it begins. Moreover, the majority of submitted concepts will have been thought up on the spot and discussed by maybe 2 or 3 other people in the thread. These concepts rarely have enough effort or discussion put into them for the TL to make a reasoned slating decision.

On average, over the past 8 projects, Concept Submissions has lasted fewer than four days. After those four days, we expect a newly elected TL to pick 6-8 concepts that will be both Competitively Viable and Community Project Viable from a pool of 50+ concepts, the vast majority of which have not been discussed by many good users. Other than those four days, we spend zero time formally discussing what makes a good concept and whether or not proposed concepts will be successful. The TL's first four days on the job are to go from "Hi! I'm your newly elected TL!" to "This is what we'll spend the next three months evaluating and attempting to accomplish." Succeed, and we get a great project. Fail, and the project and the TL's legacy go down as failures.

If the problem is too few people judging too many bad concepts in too little time, then the solution is simple: more people, fewer bad concepts, and more time. A Concept subforum open during project downtime (PR time) accomplishes all three of those solutions. It gets more people - preferably a concept QC team - looking at concepts. We get months to look at concepts before slating because this forum will be open constantly, or at least during CAP downtime. If the forum works, we will drastically raise the number of workable concepts submitted and lower the number of hasty, off-the-cuff concepts. Just as importantly, it lets users work on the competitive side of CAP at all times, even when a project isn't taking place.

(The second half of the post is below.)

The next question to answer is what that subforum should do. Before I go into detail about that, I want to touch on manpower issues. It is irresponsible to ignore manpower when discussing a new Concept subforum. CAP is not a large community and it is not significantly growing. We just had a massive PR thread about retaining our veterans. We hope that a Concept subforum will invite outsiders into the project, but we know that we do not have the manpower larger forums like C&C have. We can't start with huge projects and expect a handful of people to get it done.

Doug's post broke up QC into two phases - a "Concept Discussion" phase and a "Simulation / Play it Forward" phase. While I'm on board about the first step, I'm extremely wary of a potential Simulation phase. The idea of getting a handful of CAP elites to privately create a flash CAP to determine whether a concept is viable (and then destroy the record) is one of the most anti-CAP ideas I can imagine. Not only is it outright massive polljumping, it gives the Concept QC team an enormous leg up over the rest of the community once the concept is selected. We'd go into every CAP with a QC team that "knows what works" while the rest of the community is still figuring everything out. It's not fair to the rest of the community to have anything privately pre-decided or pre-discussed by a QC team. I also doubt we have the manpower to get that done; simulations would require a QC team to get together and flesh out dozens of concepts. That seems like far too heavy a load for a handful of QC specialists to handle. I also believe it would make the CAP far less enjoyable for the QC team: if you've already created a flash CAP with other good posters about a concept, are you really going to be that excited to spend three months doing it again with everyone?

I believe we should have a one-step QC process. We take Concepts, and evaluate whether or not they could make feasible CAPs with minimal polljumping. We can do this by comparing them to past successful and unsuccessful CAPs. Potential directions for how the concept can be played out are welcome, but formal simulations and excessive polljumping should be policed by the moderators. This is also the stage where vagueness can be eliminated so we don't have to wait for Concept Assessment to define concept-specific terms. In short, the purpose of the QC forum is to draw potential Concept Assessment conclusions to ensure that we have multiple competitively viable directions to go in if the concept were to be selected. A QC member would approve the concept once they believe it is Competitively Viable and Community Project Viable, as defined in Doug's post. It would be rejected if the QC member does not believe the concept can fit those criteria.
DougJustDoug said:
A "Viable CAP Concept" should have BOTH of the following characteristics -
  • "Competitively Viable" - Every step of the CAP process should be able to be implemented with normal competitive game mechanics and battle strategies. Mechanics and strategies should not be likely to be overpowered, underpowered, or overly contrived (ie. "gimmicks")
  • "Community Project Viable" - Every step of the CAP process should have multiple viable options to intelligently discuss, debate and decide.
The last thing I would like to touch on is what I believe should happen after a Concept is approved by QC.

Once a concept has been approved by Concept QC, it will be placed in an archive of Approved Concepts, likely a thread in the Concept subforum. Once a user's Concept has been archived or rejected, they can submit another to the QC team. The Concept Submissions thread would function as usual. Users can submit their concepts from the QC archive in the Concept Submissions thread. However, it should not be mandatory for concepts to go through QC to be slated. Users would be allowed to submit their own concepts in the same way they do now, but the OP would be updated with a warning that it's recommended to go through the QC process before submitting a concept. The stricter moderation mentioned in the Retention Issues thread will be in full force here. This way a new user would not be forced to wade through red tape in another subforum just to participate in Concept Submissions. The purpose of the Concept Subforum should be to improve, not replace Concept Submissions while minimizing polljumping and red tape.

Most importantly, the TL should have the final say on what gets slated. Period. No matter what, we cannot take that power out of the TL's hands. The TL is the primary guiding voice behind a project and it is unfair to the TL to force them to work with a Concept they do not like. To my knowledge, Plasmanta is the only CAP in which the TL did not slate the concept, and it led to friction, negativity, and a decrease in discussion quality.

tl;dr version:
  • Right now we place immense pressure on the TL to slate concepts, yet only spend a few days every project seriously discussing concepts.
  • To improve concept quality, we add a QC subforum with concept discussion, but no concept simulation.
  • Concept Simulation alienates the community and QC team, and we do not have the manpower to accomplish it anyway. Strongly opposing concept simulation.
  • One-step concept QC, in which we clean up the concept and propose potential Concept Assessments.
  • Once a concept is approved in QC, it is archived and recognized as a quality concept. A user can submit it once Concept Submissions roll around.
  • It should not be mandatory to go through Concept QC because it is not fair to new users to go through such an arduous process.
  • The TL has the final say on which concepts are slated.
 
Last edited:

ginganinja

It's all coming back to me now
is a Community Leader Alumnusis a Community Contributor Alumnusis a CAP Contributor Alumnusis a Contributor Alumnusis a Battle Simulator Moderator Alumnus
I'm just going to make a case for simulation, since I think pretty much everyone likes the idea of a Concept subforum and simulation remains the only controversial issue on the table.

Firstly, I'm going to use the example of Plasmanta a lot, because its a good example as to why simulation is useful. Under the system of DLC, where we have a concept Yes/No approval based on Dougs criteria, Plasmanta's concept actually gets recognised as a good (or 'accepted'/'approved') concept, and thats quite the issue because the concept itself wasn't actually very good, because No-one actually bothered to "play it forward".

To be fair, Plasmanta had the issue of a new TL taking over, but the primary issue of Plasmanta's concept was people coming up with innovative or different ways to use the concept, but these ideas were shut down fairly early on in the process (within the first 24 hours really) for either not being viable, or CAP Moderator rejection, and then we were left with a concept that no-one had any idea what to do with. Plasmanta prolly won the votes because it had a really cool premise, and people either banked on their new ideas to be accepted / allowed and voted for it on that premise...or they didn't "play it forward" and didn't think about how the concept would be achieved. Unfortunately, under the system DLC proposed, nothing actually changes, even if you have a favourable TL, the concept itself is "approvable", and you still end up with a concept people don't know how to implement. This however, changes if you add in simulation.

With simulation, QC can now ask "what if this", you can ask them to think ahead or "play it forward", to see if there are any potential issues within the concept that might arise instead of giving it a pass/fail grade based on how well it achieves a written definition. Like, I'm not expecting a simulation process to involve the QC mapping and planning out the entire competitive side of the CAP, but I don't think actively discouraging (let alone actively moderating) QC members to think ahead as to how a concept might function is a particularly wise decision. Its restraining, rude, and ultimately harmful to the process. Yes, ideally we want to avoid an issue where QC has "made" a CAP already, but under Dougs principal, said concept should be community viable, have multiple different avenues to explore which mitigates the issue somewhat, although I personally wouldn't be opposed to having a few QC members start giving some threads a bit of direction...they can get awfully messy in a CAP process.
 

HeaLnDeaL

Let's Keep Fighting
is an Artistis a Forum Moderator Alumnusis a CAP Contributor Alumnus
I'm going to be terribly busy over the next few weeks because of the exam crunch, but I really wanted to at least make another comment now.

I like the idea of simulations. I agree with ginganinja for the most part that running through scenarios is one of the best ways to comb through a concept and see how practical/viable it is. However, I'm somewhat confused on why we're stuck in the mentality that we can only go about these simulations in the same manner that we go about making Flash CAPs. Flash CAPs, as many have pointed out already, are incredibly time consuming and having that lengthy of a process does seem impracticable given out manpower and likely time constrictions. However, I still think simulations are important and can be done in a much more streamlined manner.

Ultimately, the way I see it, simulations are supposed to just be tools to show us that we have multiple viable options to pursue within a concept. There is no reason that we have to go through and make a whole Flash mon just to find this out. Specific, dot-by-dot details are not required to see multiple options, and therefore I don't think we should waste time in simulations filling up every single one of these dots. I think most of the steps, except perhaps typing and primary ability, can largely be generalized in order to avoid being too specific and in order to save time. Here's a quick proposal of what I envision a simulation to look like:

1) Brief Concept Assessment. This would likely supplement conversation going on in the QC thread and provide a generalized goal for the concept.

2) Typing. Exchange ideas and discuss typing options that fit the generalized goal decided from the previous step. If there's one single typing that is clearly superior, then this is a potential warning sign that the concept is too specific.

3) Ability. Potential primary abilities are discussed, yada yada. However, I'd strongly suggest only having one ability stage during simulations and have it focus on the primary ability. I think we all agree that the secondary ability should either be equal to or inferior to the primary ability, so talking only about a single ability during simulations helps streamline the process while retaining the likely better choice of abilities anyway.

4) Stat Baselines. Here, I propose that QCers/Simulators agree upon rough stat estimates rather than nitty gritty specifics. Calcs will probably be needed either way, but submitting stats that are in a range rather than as specific numbers eases the burden of finding the absolute best stat combination in such a little amount of time. Perhaps each stat should be discussed in a range of 10, so in the simulation you might end up with a stat estimate looking like 70-80 HP/110-120 Atk /130-140 Def /40-50 SAtk/60-70 SDef/80-90 Spd. Essentially, by having stat ranges we're able to give fairly restrictive stat biases without fiddling with the fine details of calcing everything imaginable.

5) Ideal Moveset. Rather than having a full movepool stage, the QCers would talk about four moves that would create the ideal moveset, without having the mon be overpowered, etc. Movepool stage is always the most extensive part of CAP, and if we focus only on a single moveset during simulations then we can save a lot of time and headaches. Sure, some CAPs are meant to have more than one viable moveset, but unless the concept is a Jack-of-all-trades type of thing then having more than four moves that make up the ideal moveset isn't super necessary during a quick simulation.

6) Repeat starting from either step 1 or step 2, depending on whether or not the brief concept assessment was too overarching, etc.

So if you guys think that a streamlined simulation process is more feasible but would like to tweak some of the steps I mentioned above, then please feel free to. Overall, I think simulations have a lot of quality checking potential in their own right, but I just don't see why the Flash CAP method is the only method.

tl;dr: Make concept simulations, but don't make them nearly as in-depth as the Flash CAP process.
 

jas61292

used substitute
is a Community Contributoris a Top CAP Contributoris a Forum Moderator Alumnusis a Battle Simulator Moderator Alumnus
I have not had the opportunity to comment here since the discussion shifted to the current topics, so I would like to come in and basically echo what DetroitLolcat said here. While I am highly in favor of a Concept Submission subforum and very much in favor of the discussion part of concept Quality Control, I am extremely opposed to the whole concept simulation idea. I could spend paragraphs talking about that, but I don't think there is much I could add that has not already been said. So, rather than reiterating everything, I'd just like to highlight one specific quote:
The idea of getting a handful of CAP elites to privately create a flash CAP to determine whether a concept is viable (and then destroy the record) is one of the most anti-CAP ideas I can imagine.
Regardless of its usefulness as a tool to determine concept viability, a formal process of simulating a concept among a select group of individuals not only is incredibly anti-community, but it also stands to ruin the upcoming projects for those who participate in them. Either you get exactly what you simulated, and the project lacks interest to you since you have done it all, or you don't, and then your entire simulation was a waste, since you clearly failed to simulate accurately. Not every CAP project is necessarily going to turn out like the concept intended. That is unfortunate, but it will always be the truth. But so long as the discussions we have are good along the way, that is not such a big deal. It should be the job of QC to make sure that concepts can support such discussions, not to try and picture concept success, because there will always be a way for someone to see project success, and many times, multiple ways. But that is not a part of what makes the concept good.

With that said, I would just like to note that while I agree with nearly all of DLC's post, I specifically would like to disagree with the comment on not making QC mandatory. While I can see it being another layer of red tape, and as such I am certainly hesitant to add it without careful consideration, I think that if it exists, it needs to be mandatory. Otherwise, I don't see it really being put to use much, and ending up being a big hassle for not much gain.
 

ginganinja

It's all coming back to me now
is a Community Leader Alumnusis a Community Contributor Alumnusis a CAP Contributor Alumnusis a Contributor Alumnusis a Battle Simulator Moderator Alumnus
Regardless of its usefulness as a tool to determine concept viability, a formal process of simulating a concept among a select group of individuals not only is incredibly anti-community, but it also stands to ruin the upcoming projects for those who participate in them. Either you get exactly what you simulated, and the project lacks interest to you since you have done it all, or you don't, and then your entire simulation was a waste, since you clearly failed to simulate accurately.
I am pretty sure that jas61292 and DetroitLolcat don't quite understand what the people who actually favour a simulation, are actually wanting. No-one is actually asking for a QC team to go in, look at each and every concept that gets submitted, then go on IRC and attempt to condense a 2-3 month process into one IRC session, with the end result being a 100% perfected CAP based on the submitted concept...and do this for EVERY concept submitted. All I am asking for, is for the QC to have permission to look at a concept and judge how effective it could actually be. Its asking for permission to actually think about how the submitted concept works and if that is something that is actually viable. You cannot, in good faith, speak to 6 (or so) quality contributors (assuming you can actually find enough people to serve on the QC), ask them to evaluate a concept as to whether it is suitable, and then order them to not think ahead as to how such a concept would run. Its roughly equivalent to asking the OU QC to judge how good a submitted analysis/set is, without playtesting it first!

It doesn't matter if the simulation fails to accurately predict the exact movepool/ability/typing/stats, because thats not what the simulation is for, its merely trying to judge how effective a concept could be, its entirely hypothetical. Nor can you claim that the process will result in a "lack of interest" among QC members, remember, each process has to be "community viable", to have multiple options available, ergo, there is no "one correct way", of running the CAP.

Lets face it, the primary reason we have a PRC topic over this is due to the influx of concepts that get slated, win polling, and then result in a clusterf**k of a thread where no-one knows what to do next. We had it in Cawmodore where we didn't know what "unusual" move to support (which resulted in the exceptionally poor choice of the binary Belly Drum), we had Volkracken, and the clusterf**K of the "partners" thread where you could only submit "uncommon options" (options that were largely uncommon due to just being bad in OU), which, ironically enough resulted in a Top 10 (iirc might have been Top 5 at the time) mon being picked...and then you have Plasmanta where no-one had any clue how to complete the winning concept. The primary issue here, is that, under DJD's criteria for a successful CAP, all of these concepts pass the basic requirements. Which means that if, we had a QC team when those Concepts were submitted, without a simulation process all of those three concepts would receive approval. With simulation, all of those concepts come under greater scrutiny. They might still be approved, I don't know, but any potential problems with each process would have been made aware of to the QC members much earlier.

I can understand if certain users feel a certain amount of dislike towards simulations, that's perfectly fine. What I do not understand, and what causes me certain amounts of frustration, is that these users are quite happy promoting a QC, which doesn't actually address the issues we have with those troublesome concepts in the past. Those prior concepts STILL get accepted, and you STILL have those same issues that resulted in this thread being made. If you dislike simulations, thats 100% fine, but please submit an actual, viable alternative that addresses the problems already outlined, because by itself, a QC with no simulation doesn't go far enough in addressing those concerns.
 

DetroitLolcat

Maize and Blue Badge Set 2014-2017
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
ginganinja said:
Lets face it, the primary reason we have a PRC topic over this is due to the influx of concepts that get slated, win polling, and then result in a clusterf**k of a thread where no-one knows what to do next. We had it in Cawmodore where we didn't know what "unusual" move to support (which resulted in the exceptionally poor choice of the binary Belly Drum), we had Volkracken, and the clusterf**K of the "partners" thread where you could only submit "uncommon options" (options that were largely uncommon due to just being bad in OU), which, ironically enough resulted in a Top 10 (iirc might have been Top 5 at the time) mon being picked...and then you have Plasmanta where no-one had any clue how to complete the winning concept. The primary issue here, is that, under DJD's criteria for a successful CAP, all of these concepts pass the basic requirements. Which means that if, we had a QC team when those Concepts were submitted, without a simulation process all of those three concepts would receive approval. With simulation, all of those concepts come under greater scrutiny. They might still be approved, I don't know, but any potential problems with each process would have been made aware of to the QC members much earlier.
All three of the examples in this post are either of a winning Concept making little sense or a Concept Assessment thread going poorly after the concept was selected. In Cawmodore and Volkraken's cases, the problem was that the Concept Assessment's conclusion left us unable to accomplish the concept. In Plasmanta's case, we ended up realizing that what sounded like an interesting concept made very little sense once submitted to closer scrutiny. Regardless, the problem was either in Concept Submissions or Concept Assessment. If the Concept Submission and Assessment go poorly, then it's likely that the CAP as a whole will go poorly. On the other hand, it's very likely that a successful Concept Assessment will lead to a successful project. Necturna, Mollux, and Malaconda all had good to great Concept Assessments, and those projects are generally considered successful. In my opinion, the only recent (Gen 5-onward) case in which the success of the Concept Assessment didn't predict the success of the project was Aurumoth (CA went well, but the project was unsuccessful for other reasons). Therefore, I believe the only part of the Concept that needs to be played forward is Concept Assessment. It's the only step that's completely imperative to a successful project. My previous post discussed one-stage QC that entails Doug's proposed QC Stage 1 plus potential Concept Assessment conclusions. If you want to break that into two steps that's fine. I see further simulation - simulation of Typing, Stats, Abilities, and Movepool - doing much more harm than good for the reasons mentioned in my previous post.

In short, a quality concept is one that can lead to a successful Concept Assessment. If we can demonstrate that a Concept is conducive to interesting, diverse Concept Assessments then we can call it a quality concept. There is no need to further simulate the concept, and I strongly oppose doing so for aforementioned reasons.

EDIT: Had a short chat with ginga about this. I don't think our opinions differ greatly, just what he sees as the "bare minimum" I see as the maximum.
[08:10] <%ginganinja> Detroit
[08:10] <%ginganinja> wanna chat
[08:10] <@DetroitLolcat> Sure. I'm writing up a response to your post in the forum at the moment anyway.
[08:11] <%ginganinja> ic
[08:11] <%ginganinja> whats it going to talk about
[08:11] <%ginganinja> I really, really hated the jas post :/
[08:12] <@DetroitLolcat> I don't think full simulations of a CAP are necessary because if you look at CAPs that have gone poorly, the problem almost always lies in Concept Submissions or Concept Assessment.
[08:12] <%ginganinja> ok I agree with that
[08:13] <%ginganinja> I don't think a full simulation is needed
[08:13] <@DetroitLolcat> As you said, Cawmodore and Volkraken went poorly because we ended up with bad conclusions in Concept Assessment (Belly Drum, Latias+Lucario core) while Plasmanta went badly because the concept seemed impossible to assess.
[08:13] <%ginganinja> I'd like it, but I can get behind not having one
[08:13] <%ginganinja> I think there needs to be a concept simulation though
[08:13] <%ginganinja> which I view as something different
[08:13] <%ginganinja> but its hard to describe in a post
[08:14] <@DetroitLolcat> Therefore, I think the only part of the process that needs to be played forward is Concept Assessment. If we have a successful concept assessment, we almost always have a successful CAP, and if we have a bad concept assessment, we almost always have a bad CAP.
[08:14] <%ginganinja> ye I can support that
[08:14] <%ginganinja> no problem
[08:15] <@DetroitLolcat> When I think "simulation", I think full playing out of Assessment, Typing, Stats, Abilities, and Movepool as suggested by Doug's post in the first page.
[08:15] <%ginganinja> yep
[08:15] <%ginganinja> I think playing it fowarf of just the concept assessment
[08:16] <%ginganinja> is the bare minimum
[08:16] <@DetroitLolcat> I think that's definitely anti-CAP as I explained in my first post. But playing Assessment forward is something I'm willing to accept because it absolutely needs to go well if we want the project to go well.
[08:16] umbreon_dan (~dan@synIRC-C0DE874B.twcny.res.rr.com) joined #cap.
[08:16] <%ginganinja> fair enough
[08:17] <@DetroitLolcat> Replace "willing to accept" with "fully support" in that previous line lol
[08:17] <%ginganinja> kk
[08:17] <%ginganinja> question:
[08:17] <@DetroitLolcat> The only time I can think of a concept assessment going well but the project going poorly is Aurumoth, while I can't think of a good project following a poor CA.
[08:17] <%ginganinja> will QC have their posts moderated
[08:18] <%ginganinja> it was touched on in a previous post
[08:18] <%ginganinja> and that ticks me off
[08:18] <%ginganinja> I'd never have done OU QC if C&C mods said they would moderate posts
[08:18] <%ginganinja> and I expect something similar would happen in CAP QC
[08:18] <%ginganinja> people dont like red tape
[08:19] <@DetroitLolcat> I don't know how to answer that at the moment. The subforum doesn't exist yet and it really hasn't been discussed rigorously in the thread.
[08:19] <%ginganinja> true
 

HeaLnDeaL

Let's Keep Fighting
is an Artistis a Forum Moderator Alumnusis a CAP Contributor Alumnus
jas61292 said:
Either you get exactly what you simulated, and the project lacks interest to you since you have done it all, or you don't, and then your entire simulation was a waste, since you clearly failed to simulate accurately.
I couldn't disagree with this mentality any more. The simulation is not about accurately predicting what the CAP would look like at all. It is about seeing if the concept is viable in a more realistic standard, which ultimately boils down to making sure a concept assessment has room for success and making sure multiple viable options are available along future steps. Furthermore, I was under the impression that simulations would be rerun to garner multiple simulated "products." This would suggest not putting all eggs in a single basket and moves away from the idea of trying to actually predict what a CAP would look like. If none of the simulations match the final outcome of a true CAP process, then I don't view that as a waste of time. If anything it speaks to the variation that the concept has.
 

Bughouse

Like ships in the night, you're passing me by
is a Site Content Manageris a Forum Moderator Alumnusis a CAP Contributor Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnus
I agree that if you get exactly what you simulated it is probably less interesting than the average CAP for the review team.

But if you get exactly what you simulated, it's sheer dumb luck. If the submission review team all get to a single simulated result, they won't recommend it to advance to the list of slatable concepts. The entire point of this is that the only concepts that go through should be both clear in what it means and diverse in how it can be done. The simulation is not a flashcap. The team is not dividing and conquering or collaborating as a CAP is typically made. Rather, each reviewer thinks up the entire CAP alone and then the reviewers compare what end results they got to.

Simulation is not about polljumping in any intentional project-influencing way. The review team just needs to confirm that what reviewers 1, 2, and 3 come up with is not identical. The CAP community will ultimately never see the irc logs of a concept that does get put in the slatable pool - only the reasons of why a concept wasn't allowed through for the TL to consider, which could either be that it's not clear enough, not viable for the metagame, or that based on the reviewer simulations, it looks like there aren't enough interesting ways to fulfill the concept.
 

nyttyn

From Now On, We'll...
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
Why don't we just compromise on this? Based on what everyone's saying, I think there's a way to just make all sides happy.

---
Compromise proposal: As a mandatory part of the concept QC process, at least 3 members of the QC team will have to simulate concept discussion, be it over conversations on the forum, IRC (preferred), or any other form of contact they can think of, so long as it is capable of keeping logs. If they feel it is necessary, the QC team may simulate further steps, but this will not be mandatory - they will simply be given the option to do so. All logs that result from the discussion and simulation of CAP Concepts must be saved and archived, and will be avalible upon request for rejected concepts, after a brief grace period granted to allow for editing, and potentially removing comments not related to concept discussion at the QC team's request (to respect the QC team's right to privacy. All that matters is the CAP, not what else they might discuss or mention). If a supermajority (2/3rds) of the participating members are satisfied, assuming no further objections, the concept will be allowed to be chosen by the TL to put on the Concept slate.
---

Making only concept assessment mandatory to simulate will satisfy the desires to minimize the amount of manpower that this will take, as well as minimize potential influence caused on the project. At the same time, the QC team will be free to put forth the extra effort, if they feel that the concept calls for it, allowing them flexibility and freedom to explore potential pitfalls of a particular project. Boom, both sides get what they want.

Also I somewhat arbitrarily chose 3 members as the minimum for discussions because 2 felt like not nearly enough, and 4 is a bit high. This number can be changed if need be. Likewise, if we feel less than a supermajority (or even more and require unanimous) for agreement is required, that can be changed too.
 
Why don't we just compromise on this? Based on what everyone's saying, I think there's a way to just make all sides happy.

---
Compromise proposal: As a mandatory part of the concept QC process, at least 3 members of the QC team will have to simulate concept discussion, be it over conversations on the forum, IRC (preferred), or any other form of contact they can think of, so long as it is capable of keeping logs. If they feel it is necessary, the QC team may simulate further steps, but this will not be mandatory - they will simply be given the option to do so. All logs that result from the discussion and simulation of CAP Concepts must be saved and archived, and will be avalible upon request for rejected concepts, after a brief grace period granted to allow for editing, and potentially removing comments not related to concept discussion at the QC team's request (to respect the QC team's right to privacy. All that matters is the CAP, not what else they might discuss or mention). If a supermajority (2/3rds) of the participating members are satisfied, assuming no further objections, the concept will be allowed to be chosen by the TL to put on the Concept slate.
---

Making only concept assessment mandatory to simulate will satisfy the desires to minimize the amount of manpower that this will take, as well as minimize potential influence caused on the project. At the same time, the QC team will be free to put forth the extra effort, if they feel that the concept calls for it, allowing them flexibility and freedom to explore potential pitfalls of a particular project. Boom, both sides get what they want.

Also I somewhat arbitrarily chose 3 members as the minimum for discussions because 2 felt like not nearly enough, and 4 is a bit high. This number can be changed if need be. Likewise, if we feel less than a supermajority (or even more and require unanimous) for agreement is required, that can be changed too.

Looks good to me. One thing I might suggest, however, is that we restrict the maximum number of the QCers working on one concept as well as the minimum. One of the big problems we're going to face going into this change is that we might get flooded with more concepts than we can handle, so tying up too many people is hardly recommendable. At this moment in time, I'm inclined to say 6 should be the maximum number of QCs that can be dedicated to one concept at a time, but that may change once we get a better idea what sort of workload we're looking at.
 

nyttyn

From Now On, We'll...
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
Looks good to me. One thing I might suggest, however, is that we restrict the maximum number of the QCers working on one concept as well as the minimum. One of the big problems we're going to face going into this change is that we might get flooded with more concepts than we can handle, so tying up too many people is hardly recommendable. At this moment in time, I'm inclined to say 6 should be the maximum number of QCs that can be dedicated to one concept at a time, but that may change once we get a better idea what sort of workload we're looking at.
nah there's no sense in limiting the QC team to having a max # of people discussing at any time. for 1 they'll likely just talk among themselves even after the simulations over/during the simulation about it regardless of what limits we try to impose making said limits 100% pointless in the first place and for 2 if only concept discussion is mandatory then things should be a lot shorter anyways so we don't have to worry too much about restrictions for the sake of time.

honestly a stronger argument for having a maximum is that more people = more discussion = more time, but even still, people are just going to ignore something as stupid as an arbitrary maximum headcount and frankly they should. the QC team is already going to have to go out of their way to get all of this done - if all of them want to discuss/sim a concept at the same time, let them, since they'll do it with or without being 'allowed' to anyways.
 

Ununhexium

I closed my eyes and I slipped away...
is a Community Contributoris a Smogon Discord Contributoris a Contributor to Smogonis a Smogon Media Contributoris a Social Media Contributor Alumnusis a Forum Moderator Alumnus
Just a quick question. Will there be a channel for QC and simulating concepts or will we just use #cap?
 

ginganinja

It's all coming back to me now
is a Community Leader Alumnusis a Community Contributor Alumnusis a CAP Contributor Alumnusis a Contributor Alumnusis a Battle Simulator Moderator Alumnus
Would prolly be better doing it in a different channel. Not only does it suddenly derail an existing conversation if its in #cap, you do have the problem of a peanut gallery chipping in when they really shouldn't. I feel kinda elitist saying that, but it is slightly true :(
 
I'm inclined to agree with Ginganinja one this one. Especially if we need to keep a record of the QC process, it would probably easier if we didn't have non-QC interference or QC members dealing with conversations not dealing with the QC process.
 

nyttyn

From Now On, We'll...
is a Forum Moderator Alumnusis a CAP Contributor Alumnus
oh i thought it was kind of assumed that it'd have its own channel. both for the reasons ginganinja stated, and because of two more reasons. 1. if we're trying to provide logs, suddenly there's a lot of random ass posts from the non QC members of #cap that need to be pruned before the logs can be posted, which adds way too much unnecessary effort. and 2. even if they don't chip in on the concept itself, if #CAP continues to chatter during the discussion/simulation, it can be seriously distracting and just flat out disruptive.

oh yeah and before i forget there's also the whole 'influence' issue that we're trying to avert/avoid. we just established that we'll only be providing logs for failed concepts to avert any potential influence on the CAP itself - why would we throw that in the shitter by then having all the discussions in public for all to see (and potentially record and share)?
 

Birkal

We have the technology.
is a Top Artistis a Top CAP Contributoris a Top Smogon Media Contributoris a Site Content Manager Alumnusis a Battle Simulator Admin Alumnusis a Senior Staff Member Alumnusis a Community Contributor Alumnus
DougJustDoug is our thread closer on this one. I think he has some final thoughts to share before concluding though; stay tuned for his post.
 

DougJustDoug

Knows the great enthusiasms
is a Site Content Manageris a Top Artistis a Programmeris a Forum Moderatoris a Top CAP Contributoris a Battle Simulator Admin Alumnusis a Smogon Discord Contributor Alumnusis a Top Tiering Contributor Alumnusis an Administrator Alumnus
Moderator
(I have had the following post in a half-done state for a couple of weeks, and I apologize for the long delay in getting it finished and posted.)

After reading the feedback to the simulations that were proposed, I agree with many of the concerns raised and my opinion has shifted a bit on the topic. I still think simulations would be a good and necessary part of vetting Concepts in a subforum, but I don't think they should be as involved as they were originally mentioned.

When I suggested simulations, my intention was that they would NOT be big, laborious, manpower-intensive things. That is why I suggested a simple 5-part format (Concept, Typing, Abilities, Stats, Movepool) that significantly compresses the CAP process when doing a simulation analysis. But even with an abbreviated format, I still pictured it as an ANALYSIS, which I guess I did see as being a fairly formal and rigorous process. When Birkal suggested we do something akin to a Flash CAP, I was think "Yeah! That's perfect!". But after reading others' take on it, now I'm not too keen on that level of detail and formalization for simulating Concepts.

To determine the best way to accomplish simulations, let's clarify some of the key goals for simulations, in terms of what we DO WANT and DON'T WANT to happen. Then we can determine the format which best achieves the positives and best avoids the negatives, knowing that we probably won't get a "perfect" format that is all good with no bad. Tradeoffs are inevitable.

Simulation Ideals

We want simulations to expose the obvious flaws in Concepts. We do not expect simulations to uncover every subtle or detailed issue.

Maybe I am undershooting the goal here, but I don't think we need to be heroes on this. We aren't trying to GUARANTEE that every CAP will have a fantastic Concept in the future. We are just trying to give CAP a BETTER chance for a good concept. Instead of thinking of this as a way to identify "the best" concepts, think of this as a way of eliminating "bad" concepts and leaving mostly "potentially good" concepts remaining.

We have a mountain of concepts to deal with in CAP, and most of them are terrible. Sticking with the mountain analogy -- we're not trying to make gold, we're just trying to mine some ore. Mining is mostly an exercise in digging through and discarding lots of worthless dirt and rock, and keeping some stuff that exhibits properties that it MIGHT have precious metal inside. *(Mineralogy nerds, please don't call me out on the science of my analogy. Hopefully you get the point, right?)

We want to force people to "play forward" a Concept in a structured, analytical manner.
This is really the biggest thing, in my opinion. Regardless of how we do it, we want to make people think about a Concept beyond just assessing if it "looks like a good idea". Even the most rigorous thinkers tend to consider Concepts holistically when judging if they are good or bad. People make numerous assumptions and take all sorts of logical shortcuts in their thinking, and don't really realize they are doing so. I know I am guilty of this, and I am generally pretty good at thinking in a structured way.

By forcing everyone to consider Concepts in a structured, sequential manner and to commit to discreet outcomes in each step of their thinking -- that's how we get real value from a simulation. If simulations are relatively unstructured, they will be little more than "Think about this concept and post your impressions". This will encourage the same holistic thinking we always get with concept commentary, and really won't expose any flawed concepts beyond what we are capable of exposing already. A structured approach doesn't guarantee anything, but it should help us look below the surface of the concept, which would be a good thing.

We want simulations to encourage autonomous contribution by QCers.
If we require too much or too close interaction between members to play forward a Concept, we'll cause more problems than we solve. We don't want multiple people to have to get together offline to discuss concepts, because it will decrease transparency for the rest of the project (anti-community = bad).
If we require people to go back and forth with each other in some convoluted forum posting process, it will take way too long with participants all over the globe and long lag times between interactions (longer processes = bad).
If we encourage small groups of knowledgeable users to do their own little mini-CAPs amongst themselves to vet concepts, we'll end up with know-it-all posters in the regular CAP project who have already pre-built the CAP amongst themselves beforehand (annoying elite cliques = bad).

There are probably ways to mitigate all these problems, but we can avoid all that if we make simulations largely independent exercises by individuals. The result of simulation should be a single forum post in the subforum (ie. "I have done a simulation of the XYZ concept, here it is: <simulation in whatever format we deem appropriate>). Others can comment on the simulation, point out issues, etc -- but the creation of the simulation itself should mostly be an individual exercise, in order to avoid the problems mentioned above.


Simulation Implementation
I think HeaLnDeaL is on the right track in terms of how a relatively simple concept simulation could be structured. Here is a proposed structure to consider:

A simulation would consist of a single post with five parts/steps - Assessment, Typing, Ability, Stats, Moves.
Each step of the simulation should do the following:

1) Explain multiple viable high-level options that could reasonably be considered by the community, based on the choices of previous steps. The simulation step should illustrate that the community would have more than one interesting, competitively-viable option to discuss and decide.

2) Pick an option to add to the foundation for the next simulation step. The option chosen should be the one most likely to be chosen by the community in a real CAP. This is a subjective call, but should be based on reason as much as possible. The point is not to accurately predict the future or read peoples' minds. Each step simply needs a discreet outcome to serve as a basis for the next step, and these outcomes should not be random or a whim.

Assessment - Mention a few high-level directions the project could take to satisfy the concept.
Typing - Present a few typing options that are reasonably representative and diverse, probably should acknowledge CAP's heavy historical bias for rare or unique typing, etc.
Ability - Like HealnDeal mentioned, only the primary ability needs to be considered for simulation. Abilities that achieve similar general competitive goals for the concept should not be presented as "multiple options".
Stats - No need to quote specific stat lines here, general build descriptions and bias phrases are fine. If specific stats need to be referenced, they should be ranges.
Moves - Present the key moves and controversial moves of the main movesets that would be viable for the concept. Since Moves is the last competitive step, it is not necessary to choose an outcome. Simply simulate that a vibrant discussion is reasonably likely.

Discussing/Critiquing Simulations
Critiquing simulations is how multiple people can "work together on a simulation", with the goal of identifying narrow or dead-end concepts. A narrow concept is one with limited choices at some point. A dead-end concept is one that has no viable options at some point. All projects tend to become more narrow towards the end, so we are really looking for concepts that narrow or dead-end too early in the process.

Commenters should critique the content and reasoning of posted simulations. Point out options that were not considered, and why they would are not legitimately competitively-viable or not likely to have meaningful intelligent support. Also point out options that would have highly polarized voting support, independent of competitive logic (ie. options that don't have a chance in hell or options that will win by a landslide in public poll, regardless of what the "intelligent people" think).

Commenters may present alternative outcomes from certain steps in order to illustrate dead ends that are not obvious in the simulation, or to illustrate increased variety of choices than what was presented in the simulation. Commenters should not effectively rewrite the entire simulation in the course of a critique. If the commenter thinks the simulation completely missed the mark, they should just do a simulation of their own and post it as such.


Manpower and Timing
I am definitely concerned about whether CAP will have the manpower to do something like this, particularly in a dedicated subforum outside the normal workflow of an ongoing CAP process. And considering the other issues we have right now, in terms of starting the next CAP project in the wake of ORAS, this may not be something we should try to do right now. Or maybe it is EXACTLY what we need to do in order to give the next CAP a better rudder, in the face of ORAS-related uncertainty.

But is the format mentioned above really all that much work? It's basically asking people to think about a concept and make a single post with a certain logical structure. That isn't a whole lot of work, although admittedly, it took me weeks to get this post completed, so I can't say a single post is "no big deal". nyttyn presented some compelling reasons that detailed Flash-CAP-like simulations would never work. Does a simpler, more individual effort make it more feasible to accomplish?

BTW, I am assuming simulations would only be done for concepts that have made it past a high-level vetting process whereby a QC team (or other process) sifts through all the Concepts and proposes which ones should be simulated. Later we can work out the exact mechanics of the pre-vetting process to ensure it is both efficient and fair, with checks and balances and all that. Right now, my main concern is whether we can pull off a more structured way of validating concepts via simulations of some kind.
 

ginganinja

It's all coming back to me now
is a Community Leader Alumnusis a Community Contributor Alumnusis a CAP Contributor Alumnusis a Contributor Alumnusis a Battle Simulator Moderator Alumnus
I don't think simulations really need to be overly complicated. All you need to do is have a thread the simulation peeps can post in, a smogon conversation for peeps to privately discuss various concepts (bypasses the "timezone issue with IRC), and lastly, an IRC channel thats available for real time discussion among online peeps. If its a particularly controversial concept (ie peeps are split), then an IRC session can be scheduled to hash it out. When a decision is reached, you have a peep post in the thread with "APPROVED", or "REJECTED" and you move on. I think this can be partiarly informal, when I was on the OU QC simulations was what you could do all the time (heck, most competent OU players can visualise a concept, build a team and then mentally form an idea as to what its strengths / weaknesses / is it viable etc in a short timeframe), you would think about it, and judge how successful it would be, and CAP could adopt a similar train of thought. I really, don't think there needs to be an in depth, scientific process involved with simulations, you either know if its good or not, and I know many people would hate having to write an essay explaining what they already know, that Concept X is Good/Bad.
 

Users Who Are Viewing This Thread (Users: 1, Guests: 0)

Top