Ritalincel
💢
- Joined
- Oct 20, 2018
- Posts
- 89,837
- Reputation
- 146,799
For several years now, we have been steadily scaling up our safety enforcement mechanisms. In the early phases, this involved addressing reports across the platform more quickly as well as investments in our Safety teams, tooling, machine learning, etc. – the “rising tide raises all boats” approach to platform safety. This approach has helped us to increase our content reviewed by around 4x and accounts actioned by more than 3x since the beginning of 2020. However, in addition to this, we know that abuse is not just a problem of “averages.” There are particular communities that face an outsized burden of dealing with other abusive users, and some members, due to their activity on the platform, face unique challenges that are not reflected in “the average” user experience. This is why, over the last couple of years, we have been focused on doing more to understand and address the particular challenges faced by certain groups of users on the platform. This started with our first Prevalence of Hate study, and then later our Prevalence of Holocaust Denialism study. We would like to share the results of our recent work to understand the prevalence of hate directed at women.
The key goals of this work were to:
We specifically wanted to look at how this hateful content is impacting women-oriented communities, and users perceived as being women. We used a manually curated list of over 300 subreddits that were women-focused (trans-inclusive). In some cases, Redditors self-identify their gender (“...as I woman I am…”), but one the most consistent ways to learn something about a user is to look at the subreddits in which they participate.
For the purposes of this work, we will define a user perceived as being a woman as an account that is a member of at least two women-oriented subreddits and has overall positive karma in women-oriented subreddits. This makes no claim of the account holder’s actual gender, but rather attempts to replicate how a bad actor may assume a user’s gender.
With those definitions, we find that in both women-oriented and non-women-oriented communities, approximately 0.3% of content is identified as being hateful content directed at women. However, while the rate of hateful content is approximately the same, the response is not! In women-oriented communities, this hateful content is nearly TWICE as likely to be negatively received (reported, downvoted, etc.) than in non-women-oriented communities (see chart). This tells us that in women-oriented communities, users and mods are much more likely to downvote and challenge this kind of hateful content.
Title: Community response (hateful content vs non-hateful content)
Next, we wanted to see how users respond to other users that are perceived as being women. Our safety researchers have seen a common theme in survey responses from members of women-oriented communities. Many respondents mentioned limiting how often they engage in women-oriented communities in an effort to reduce the likelihood they’ll be noticed and harassed. Respondents from women-oriented communities mentioned using alt accounts or deleting their comment and post history to reduce the likelihood that they’d be harassed (accounts perceived as being women are 10% more likely to have alts than other accounts). We found that accounts perceived as being women are 30% more likely to receive hateful content in response to their posts or comments in non-women-oriented communities than accounts that are not perceived as being women. Additionally, they are 61% more likely to receive a hateful message on their first direct communication with another user.
Finally, we want to look at Reddit Inc’s response to this. We have a strict policy against hateful content directed at women, and our Rule 1 explicitly states: Remember the human. Reddit is a place for creating community and belonging, not for attacking marginalized or vulnerable groups of people. Everyone has a right to use Reddit free of harassment, bullying, and threats of violence. Communities and users that incite violence or that promote hate based on identity or vulnerability will be banned. Our Safety teams enforce this policy across the platform through both proactive action against violating users and communities, as well as by responding to your reports. Over a recent 90 day period, we took action against nearly 14k accounts for posting hateful content directed at women and we banned just over 100 subreddits that had a significant volume of hateful content (for comparison, this was 6.4k accounts and 14 subreddits in Q1 of 2020).
Measurement without action would be pointless. The goal of these studies is to not only measure where we are, but to inform where we need to go. Summarizing these results we see that women-oriented communities and non-women-oriented-communities see approximately the same fraction of hateful content directed toward women, however the community response is quite different. We know that most communities don’t want this type of content to have a home in their subreddits, so making it easier for mods to filter it will ensure the shithead users are more quickly addressed. To that end, we are developing native hateful content filters for moderators that will reduce the burden of removing hateful content, and will also help to shrink the gap between identity-based communities and others. We will also be looking into how these results can be leveraged to improve Crowd Control, a feature used to help reduce the impact of non-members in subreddits. Additionally, we saw a higher rate of hateful content in direct messages to accounts perceived as women, so we have been developing better tools that will allow users to control the kind of content they receive via messaging, as well as improved blocking features. Finally, we will also be using this work to identify outlier communities that need a little…love from the Safety team.
As I mentioned, we recognize that this study is just one more milestone on a long journey, and we are constantly striving to learn and improve along the way. There is no place for hateful content on Reddit, and we will continue to take action to ensure the safety of all users on the platform.
The key goals of this work were to:
- Understand the frequency at which hateful content is directed at users perceived as being women (including trans women)
- Understand how other Redditors respond to this content
- Understand how Redditors respond differently to users perceived as being women (including trans women)
- Understand how Reddit admins respond to this content
We specifically wanted to look at how this hateful content is impacting women-oriented communities, and users perceived as being women. We used a manually curated list of over 300 subreddits that were women-focused (trans-inclusive). In some cases, Redditors self-identify their gender (“...as I woman I am…”), but one the most consistent ways to learn something about a user is to look at the subreddits in which they participate.
For the purposes of this work, we will define a user perceived as being a woman as an account that is a member of at least two women-oriented subreddits and has overall positive karma in women-oriented subreddits. This makes no claim of the account holder’s actual gender, but rather attempts to replicate how a bad actor may assume a user’s gender.
With those definitions, we find that in both women-oriented and non-women-oriented communities, approximately 0.3% of content is identified as being hateful content directed at women. However, while the rate of hateful content is approximately the same, the response is not! In women-oriented communities, this hateful content is nearly TWICE as likely to be negatively received (reported, downvoted, etc.) than in non-women-oriented communities (see chart). This tells us that in women-oriented communities, users and mods are much more likely to downvote and challenge this kind of hateful content.
Title: Community response (hateful content vs non-hateful content)
| Women-oriented communities | Non-women-oriented communities | Ratio |
|---|---|---|---|
Report Rate | 12x | 6.6x | 1.82 |
Negative Reception Rate | 4.4x | 2.6x | 1.7 |
Mod Removal Rate | 4.2x | 2.4x | 1.75 |
Finally, we want to look at Reddit Inc’s response to this. We have a strict policy against hateful content directed at women, and our Rule 1 explicitly states: Remember the human. Reddit is a place for creating community and belonging, not for attacking marginalized or vulnerable groups of people. Everyone has a right to use Reddit free of harassment, bullying, and threats of violence. Communities and users that incite violence or that promote hate based on identity or vulnerability will be banned. Our Safety teams enforce this policy across the platform through both proactive action against violating users and communities, as well as by responding to your reports. Over a recent 90 day period, we took action against nearly 14k accounts for posting hateful content directed at women and we banned just over 100 subreddits that had a significant volume of hateful content (for comparison, this was 6.4k accounts and 14 subreddits in Q1 of 2020).
Measurement without action would be pointless. The goal of these studies is to not only measure where we are, but to inform where we need to go. Summarizing these results we see that women-oriented communities and non-women-oriented-communities see approximately the same fraction of hateful content directed toward women, however the community response is quite different. We know that most communities don’t want this type of content to have a home in their subreddits, so making it easier for mods to filter it will ensure the shithead users are more quickly addressed. To that end, we are developing native hateful content filters for moderators that will reduce the burden of removing hateful content, and will also help to shrink the gap between identity-based communities and others. We will also be looking into how these results can be leveraged to improve Crowd Control, a feature used to help reduce the impact of non-members in subreddits. Additionally, we saw a higher rate of hateful content in direct messages to accounts perceived as women, so we have been developing better tools that will allow users to control the kind of content they receive via messaging, as well as improved blocking features. Finally, we will also be using this work to identify outlier communities that need a little…love from the Safety team.
As I mentioned, we recognize that this study is just one more milestone on a long journey, and we are constantly striving to learn and improve along the way. There is no place for hateful content on Reddit, and we will continue to take action to ensure the safety of all users on the platform.
