Surveys have recently been circulated on the Stratego Forum. Their intention has been to establish how the community feels about certain topics, but without discussing ‘How big is the Stratego Forum community?' the significance of any survey result is difficult to assess. Perhaps the opinions of a few people may only be regarded as a drop in the ocean, or given too much credence, or are assigned too much weight that is disproportionate to the size of the community.
In the recent draft proposal by MT* (to amend the terms relating to the removal of a moderator), the pre-requisite to any community poll was that 25 signatures were required within a week. Is that clause grounded in logic and borne out by the actual numbers?
The aim of this analysis is to establish what levels of numbers of the Stratego Forum Community constitute significance (with particular reference to MT’s proposal) because without such an assessment it is impossible to interpret results and make decisions with confidence.
Let me preface this next section by mentioning that you do not need to take my word for any numbers that appear - all the Forum statistics presented in this analysis are searchable and verifiable here: http://forum.strateg...desc&filter=ALL
How big is the Forum community?
To answer simply, there are 3089 Forum members. This is the total historical number of accounts who have ever joined the Forum.
Membership of a gym does not guarantee that you are lifting weights every daily. You may have last visited the gym 5 years ago. When we are asking about the size of the Forum Community the question we really need to ask is ‘How big is the active Forum community?’ It is of no value to include, for example, an account which joined 3 years ago and never posted. So we need to apply a sensible parameter to this membership to see who actually visits this gym.
‘Forum Activity’ is the most viable parameter and presents a few more choices – as well as determining when a member joined, we can see when they were last online, how many posts they have made, and how recently they last posted.
Members Filtered by Activity
Let us begin with counting the Members who have visited the Forum across different time periods.
1. Table of Members who visited the Forum in the last x months
394 Members who visited the Forum at least once in the last 6 months
365 Members who visited the Forum at least once in the last 5 months
321 Members who visited the Forum at least once in the last 4 months
278 Members who visited the Forum at least once in the last 3 months
217 Members who visited the Forum at least once in the last 2 months
167 Members who visited the Forum at least once in the last 1 month
Now, this table includes anyone who has visited the Forum, regardless of their level of involvement in the community: they needn’t necessarily have made a post, they may have just logged in once and decided it wasn’t for them, or they may have become a die-hard member. It seems logical to apply some level of activity to distinguish a passing tourist from a resident of the city. After all, if we were to gauge the community reaction to opening a local cinema in Patras, we would not seek to consult the population of Tennessee – the breadth of the sample must be relevant. Luckily there is an available filter to measure whether a Forum member is an active participant – the number of Forum Posts. The following table does not require a Forum member to be particularly active – just one post within the time period in question would suffice
2. Table of Members making 1 Post or more
271 Members making 1 post or more in the last 6 months
220 Members making 1 post or more in the last 5 months
217 Members making 1 post or more in the last 4 months
196 Members making 1 post or more in the last 3 months
153 Members making 1 post or more in the last 2 months
120 Members making 1 post or more in the last 1 month
It is important at this stage to recognize that the size of community is dependent on the Time Period. ‘How big is the Active Community over a period of 6 months?’ is a very different question from ‘How big is the Active Community over a month?’
If we are to measure the Active Community, we should apply a minimum post count. A member making just one post historically can hardly be considered a full-time community member. To avoid a number that anyone is likely to argue against, let us apply a very small minimum total post count of 5. Let us refer to members who have a total post count of 5 or more as Basic Members.
3. Table of Basic Members (Total Post Count > 4) making 1 Post or more
172 Basic Members making 1 post or more in the last 6 months
163 Basic Members making 1 post or more in the last 5 months
153 Basic Members making 1 post or more in the last 4 months
141 Basic Members making 1 post or more in the last 3 months
118 Basic Members making 1 post or more in the last 2 months
98 Basic Members making 1 post or more in the last 1 month
Now let us look at a clause in MT’s proposal. For a petition to reach the polling stage, 25 signatures are required in one week.
While we can drop the condition that requires a member to have actually made a post in the time period specified, it is important that the member has visited the Forum in that time for a very simple reason – if a member does not visit the Forum during a week in which a poll, survey, or petition is published, then that member will have no awareness of such matters and cannot be legitimately counted as currently active
4. Table of Basic Members (Total Post Count > 4) visiting the Forum
197 Basic Members who have visited the Forum in the last 6 months
185 Basic Members who have visited the Forum in the last 5 months
177 Basic Members who have visited the Forum in the last 4 months
169 Basic Members who have visited the Forum in the last 3 months
150 Basic Members who have visited the Forum in the last 2 months
125 Basic Members who have visited the Forum in the last 1 months
115 Basic Members who have visited the Forum in the last 3 weeks
105 Basic Members who have visited the Forum in the last 2 weeks
96 Basic Members who have visited the Forum in the last 1 week
Another clause proposed by MT is that for a voter to become eligible for the poll they must have made a minimum of 50 posts by the beginning of the petition week. The logic behind this clause is clear. MT do not want people making fake accounts to bolster a vote, or perhaps recruiting their friends to vote – they only want the opinion of those who care (i.e. the active community).
Let us now apply the 50 post criteria to the data. Let us refer to members with a total post count of 50 or more as Eligible Members.
5. Table of Eligible Members (Total Post Count > 49) visiting the Forum in the last x weeks
80 Eligible Members who have visited the Forum in the last 1 month
74 Eligible Members who have visited the Forum in the last 3 weeks
69 Eligible Members who have visited the Forum in the last 2 weeks
65 Eligible Members who have visited the Forum in the last 1 week
As you can see, when you require that a member can only vote when they have a total post count of 50 or more, this reduces the number of available members for any time-frame by approximately 33% – however, we now have a big inconsistency. Under MT’s proposal, anyone is allowed to be involved in the petition phase, but a completely different set of people are allowed to be involved in the polling stage. The logical question to ask is: ‘Why?!’ The qualifying criteria for each stage must be consistent. If not, it is like asking a room of people what they want to eat for lunch, but then only counting the opinion of those who have a beard.
There is a further condition in MT’s proposal that Eligible Members must also have signed the Alias Register. You might argue that signing the Alias Register isn’t a great obstacle to those wishing to vote, but it does have the potential to further reduce the number of Eligible Members, particularly when a sizeable portion have not signed the Alias Register. Let us refer to those members who have signed the Alias Register as Registered Eligible Members.
6. Table of Registered Eligible Members (Total Post Count > 49) visiting the Forum in the last x weeks
58 Registered Eligible Members who have visited the Forum in the last 1 month
54 Registered Eligible Members who have visited the Forum in the last 3 weeks
50 Registered Eligible Members who have visited the Forum in the last 2 weeks
49 Registered Eligible Members who have visited the Forum in the last 1 week
Of course, we do need to exclude MT members and honorary members who are hardly likely to risk a relationship and betray a colleague to join a petition against their own team. Let us refer to those members as Authenticated Registered Eligible Members. These members actually account for 7 of the total for each line, and must not be counted, thus:
7. Table of Authenticated Registered Eligible Members (Total Post Count > 49) visiting the Forum in the last x weeks
51 Authenticated Registered Eligible Members who have visited the Forum in the last 1 month
47 Authenticated Registered Eligible Members who have visited the Forum in the last 3 weeks
43 Authenticated Registered Eligible Members who have visited the Forum in the last 2 weeks
42 Authenticated Registered Eligible Members who have visited the Forum in the last 1 week
In other words, should a poll be open for a month only to members who have signed the Alias Register and have a total post count of 50 or more, then we could expect 51 people to have the potential to participate. I stress the word ‘month’ because it is important that the timescale is matched to the time available to record opinion. If the time available to take a poll was only two weeks then this would decrease the size of the eligible community (in regards to MT’s proposal) to 43. Now, I stress the word ‘potential’ because there is a further consideration – you cannot operate under the assumption that every member who has visits to the Forum would participate in a poll or survey.
The reality is that many simply will not care enough to venture an opinion. Perhaps they only visit the Forum to participate in a tournament PM, or to visit a certain section of the Forum. Having 50 members visiting the Forum in the past week does not mean that you can expect 50 members to participate in a survey or poll that has a time-frame of one week. In fact, of the many polls taken historically on this site, it is extremely rare for a poll (opened for a time far longer than a month) to reach even 20 participants – and polls ask simple questions demanding far less time than a survey.
Given that an unknown proportion of these members will not care enough to participate, how can we estimate this proportion?
We can compare the numbers visiting the Forum who do not post in that time period (bystanders) with the numbers who do make a post in the time period (activists). Table 3 shows Basic Members posting in the Forum whilst Table 4 shows Basic Members visiting the Forum. We can therefore calculate the percentage of bystanders and activists respectively as follows:
10. Table displaying proportion of Basic Members who post (activists) and those who do visit without posting (bystanders)
197 Basic Members visited the Forum in the last 6 months (of which 172 posted) Bystanders = 12.6%
185 Basic Members visited the Forum in the last 5 months (of which 163 posted)
Bystanders = 10.8%
177 Basic Members visited the Forum in the last 4 months (of which 153 posted)
Bystanders = 13.6%
169 Basic Members visited the Forum in the last 3 months (of which 141 posted)
Bystanders = 16.6%
150 Basic Members visited the Forum in the last 2 months (of which 118 posted)
Bystanders = 21.3%
125 Basic Members visited the Forum in the last 1 month (of which 98 posted)
Bystanders = 21.6%
In any given period therefore, between 10-20% of Basic Members who visit the Forum will not get actively involved.
We can also look at the proportion of bystanders and activists among Eligible Members (those with 50 total posts or more):
11. Table displaying proportion of Eligible Members who post (activists) and those who visit without posting (bystanders)
102 Eligible Members visited the Forum in the last 6 months (of which 85 posted) Bystanders = 16.7%
98 Eligible Members visited the Forum in the last 5 months (of which 83 posted)
Bystanders = 15.3%
95 Eligible Members visited the Forum in the last 4 months (of which 81 posted)
Bystanders = 14.7%
93 Eligible Members visited the Forum in the last 3 months (of which 77 posted)
Bystanders = 17.2%
90 Eligible Members visited the Forum in the last 2 months (of which 73 posted)
Bystanders = 18.9%
80 Eligible Members visited the Forum in the last 1 month (of which 63 posted)
Bystanders = 21.3%
This gives a similar result, with between 15-20% of Eligible Members visiting the Forum whilst not getting actively involved.
I will end by addressing and attempting to answer 10 important questions that should be asked when trying to obtain a significant and meaningful result. All of the following questions should be approached in the context of the size and activity of the Stratego Community.
Question 1 - What format should a poll take?
This can either be posted to the Forum or sent as a PM. A PM has two clear advantages - the sender can see if the message is received and viewed (this is not possible in a Forum thread) and the recipient can also be targeted. Rather than relying on the arbitrary nature of a Forum thread (some members may not see the thread - and over time the thread becomes buried) the PM method makes it possible to directly approach all Eligible members. The sender of the poll needs only to send out a few group PMs asking for an opinion, and all doubt as to whether the recipient is aware is instantly cast aside, thus giving far more validity to the PM method.
Verdict: Poll by PM to all Eligible Members (sent by MT)
Question 2 - How long should a poll stay open?
A poll conducted by PM can stay open far longer than a poll in a Forum thread - and still maintain authenticity. The longer the time-frame of a poll, the more members will be aware of its existence. To illustrate this, a poll open for only one day will only capture the opinions of a very small sample.
On the other end of the spectrum, it is undesirable for a poll to remain open for too long. MT understandably does not want such an event to drag on endlessly - closure is required. This is sound logic for setting the length of a poll for one month. From a perspective of authenticity and reaching the community - it should certainly be no shorter than one month.
Verdict: One month
Question 3 - Who should be eligible to take part in a poll?
Eligibility is necessary to ensure that the member cares enough about the community for their result to be considered. I understand MT wanting to keep this number as high as 50 (for reasons explained earlier - so that fledgling members cannot be persuaded to vote by a friend), but this is to venture that the opinion of someone with a post count of 50 is much more relevant than someone with a post count of 45.
Of course - you can make the same argument wherever that number lies, hence why this section is the most subjective – there is no number of posts that indicates that a member is invested in the community. The criteria should be high enough only to establish that a member cares and is invested in the community on some level.
Personally I would suggest a two factor approach to eligibility.
1. Members who have a total post count of X posts or more are Eligible
2. They have made at least one post in the Forum in the three months prior to the opening of the Poll
In short, the first condition guarantees that the Member is invested in the community, whilst the second condition ensures the member is currently active and prevents dormant users being brought out of the shadows to vote on the request of a friend.
Additionally with the second condition there is no need to require the signing of the Alias Register
Verdict: Eligibility based on Forum activity is viable, provided that the eligibility criteria for the poll and petition are consistent
Question 4 - How many participants of a poll are required to consider the results meaningful?
First let us estimate the number of Members who would have their vote counted. This depends on the Eligibility Criteria established in Question 3. For the sake of argument, let us use the criteria that MT proposed - which is that members are only eligible with a total post count of 50 or more, and let us also add Condition 2. from Question 3. (Members must have made at least one post in the Forum in the past three months). If these were the parameters for eligibility, how many Eligible Members would there be?
The answer begins by using Table 5. There are 80 Eligible Members who have visited the Forum in the last month. Of these 80, 3 have not made a post in the last 3 months.
This brings our total of Eligible Members to 77.
The next stage is unavoidably subjective. You could argue that there should be no minimum number of participants. After all, if a poll has been sent to every eligible member by PM - and they have had a month to respond, what basis is there to assert that a certain level of participation is not enough?
I understand that MT would wish to avoid the scenario where a poll was sent out to all eligible members - and only 20 cared enough to respond. But is that grounds to arbitrarily state that the poll is to be negated through lack of participation? You cannot count abstinence as an expression of either support or disapproval.
If 11% of the community support an idea, 9% of the community disapprove of the idea, and 80% abstain completely - surely the relevant factor is whether they have been given the opportunity to provide an opinion. Under democratic principles at least - the example above would be supported.
Verdict: Any minimum number of participants is entirely subjective. What is of more importance is that all Eligible Members - whatever the criteria - are invited to give their opinion
Question 5 - What % of approval should be required for a result to be determined?
MT’s proposal suggested that 50% (+1 vote) is enough to determine a decision either way. This is in line with Democratic principles, and there is no reason to suggest anything different
Verdict: 50% plus 1 vote
Question 6 - Should a poll require a petition as a pre-requisite?
The logic behind requiring a petition as a pre-requisite to a poll is sensible - it determines whether there is enough support for a poll to be justified. (There is little point sending out a poll to the entire community if only a couple of people would support the petition).
Question 7 - What format should a petition take?
The format of the petition should be a Forum thread for one simple reason. If the petition is sent out by PM to eligible members then you might as well not bother with a poll. (You would be wasting Members time by asking them the same question twice).
Verdict: Forum Thread
Question 8 - How long should a petition stay open?
As previously discussed in Question 1, Forum threads quickly become buried and do not have the guarantees of reaching recipients compared to PMs.
We can assert with reasonable confidence that any member that visits the Forum on the day the petition is published will become aware of the petition. We cannot assert that any member who visits the Forum 2 weeks after the petition is published will see the petition.
So due to the fact that the visibility of a petition decays with time, we should be answering this question with regard to how many Eligible Members we can expect to see the petition. If a one month poll should seek the opinions of 77 Eligible Members (Question 4), then the petition needs to be open for a time long enough that the majority of these Eligible Members will become aware of it.
Based again on the numbers in Table 5, if the petition was open for two weeks then we could estimate that as many as 69 Eligible Members may see the petition. Of course a proportion of them would not, but this time period is long enough to raise awareness amongst the majority of Eligible Members. You could make the petition longer, but an additional week – due to the decaying visibility - would likely not increase the number of Eligible Members that you would reach.
If the petition duration is too short, then the number of Eligible Members who will become aware of the petition’s existence will be too small a proportion.
Verdict: Two Weeks
Question 9 - Who should be eligible to take part in a petition?
Verdict: The same members who would be eligible to take part in a poll on the same subject
Question 10 - How many signatures should be required for a petition to warrant a poll?
The number of signatures required should match the time-frame in question, and be consistent with the number of Eligible Members.
We can estimate the number of Eligible Members who will become aware of the petition. If the petition is only open for a week, then the number of Eligible Members will be the number of Eligible Members who have visited the Forum in the past week. This number is provided in Table 5 - it is 65.
Now we can assume that a reasonable number of those 65 will become aware of the petition. As mentioned earlier in the discussion below Table 7, there will of course be some who visit the Forum for a specific purpose and do not take the time to browse the Forum and become aware of the petition.
In addition, the analysis of Table 11 suggests that there will also be 15-20% of Eligible Members who do not become actively involved in such matters. This will reduce the number of Eligible Members that we can expect to consider a petition significantly. We cannot assume that users of the Forum will participate - whatever their opinion. Members may feel pressure to vote one way or another, or may prefer to abstain entirely
Furthermore, as mentioned before, historical surveys and polls have rarely ever reached 25 total participants - let alone 25 participants who agree with each other.
So how do you determine the number of those members that should sign a petition in order for it to succeed? Interpreting a poll is far easier - you can just draw the line down the middle at 50%. But there is a difference that makes comparing polls and petitions difficult - a poll provides the results of those who are for and against a proposal, whilst a petition provides only one half of the numbers. As such you cannot use a %, you must use a discrete number. How do you determine that number?
I think the most important factor to consider is the purpose of a petition - to assess whether there might be support for a poll. It is quite feasible that only 5 people sign a petition and then 50 members provide support in a poll. It is also feasible that 30 people sign a petition and then only the same 30 provide support in a poll.
If the ultimate aim of a petition is to assess potential support for a poll, then one must accept the possibility that any minimum number of signatures might actually - for the reasons discussed above (eg Members lacking awareness of the petition and members preferring to avoid public conflict) - derail a petition that would have succeeded.
To anyone who states ‘there must be 25 signatures’ I would counter: On what possible grounds? As the tediously long answer to this question reveals - this is just far too subjective, and there are too many factors at play.
What is certain is that the number of required signatures should be high enough to avoid wasting everyone’s time, yet low enough to ensure that a potentially successful petition is not derailed.
A more sensible approach to adopt is to consider the following scenario:
A survey asks the Community to rate the performance of Member X. The options are Excellent, Good, Satisfactory, Terrible and Poor.
The results of the survey reveal that 10 people rate Member X positively (either Excellent, Good or Satisfactory) and 15 people rate Member X negatively (either Terrible or Poor).
If a petition proposed to remove Member X, we can estimate that at least 10 people would not support the petition. Therefore - in the context of this particular question - the minimum number of required signatures should be 11.
Verdict: The number of required signatures for a petition to succeed should be the minimum number which means the result of a poll could not be guaranteed.