Content Moderation

​

What is Content Moderation?

Content moderation reviews content to ensure it meets certain standards or guidelines and flags violations. This can include identifying content that may be inappropriate, offensive, or harmful.

The path to use content moderation:

core/content-moderation

​

Content Moderation Topics

By default, content moderation flags for the following topics:

• Accidents: Any man-made incident that happens unexpectedly and results in damage, injury, or death.
• Alcohol: Content that discusses any alcoholic beverage or its consumption.
• Crime Violence: Content that discusses any type of criminal activity or extreme violence that is criminal in nature.
• Drugs: Content that discusses illegal drugs or their usage.
• Gambling: Gambling on casino-based games such as poker, slots, etc. as well as sports betting.
• Hate Speech: Content that’s a direct attack against people or groups based on their sexual orientation, gender identity, race, religion, ethnicity, national origin, disability, etc.
• Health Issues: Content that discusses any medical or health-related problems.
• Marijuana: Content that discusses marijuana or its usage.
• Natural Disasters: Phenomena that happens infrequently and results in damage, injury, or death. Such as hurricanes, tornadoes, earthquakes, volcano eruptions, and firestorms.
• Negative News: News content with a negative sentiment which typically occur in the third person as an unbiased recapping of events.
• NSFW (Adult Content): Content considered “Not Safe for Work” and consists of content that a viewer would not want to be heard/seen in a public environment.
• Pornography: Content that discusses any sexual content or material.
• Profanity: Any profanity or cursing.
• Sensitive Social Issues: Content that may be considered insensitive, irresponsible, or harmful to certain groups based on their beliefs, political affiliation, sexual orientation, or gender identity.
• Terrorism: Terrorist acts as well as terrorist groups. Examples include bombings, mass shootings, and ISIS. Note that many texts corresponding to this topic may also be classified into the crime violence topic.
• Tobacco: Content that discusses tobacco and tobacco usage, including e-cigarettes, nicotine, vaping, and general discussions about smoking.
• Weapons: Content that discusses any type of weapon including guns, ammunition, shooting, knives, missiles, torpedoes, etc.

​

Running Content Moderation

To run content moderation:

resp = moonshine.run(
  flow="core/content-moderation",
  index="your-index",
  video_id="[file_id].mp4"
)

Content moderation only runs on one video at a time.

​

Limiting Moderation Topics

To limit the topics that are moderated, you can use the topics parameter. For example, to only moderate content related to drugs, you can use the following code:

resp = moonshine.run(
  flow="core/content-moderation",
  index="your-index",
  video_id="[file_id].mp4",
  topics=["drugs"]
)

​

Using Custom Prompts

To use a custom moderation prompt:

resp = moonshine.run(
  flow="core/content-moderation",
  index="your-index",
  video_id="[file_id].mp4",
  prompt = "description of custom moderation"
)

​

Content Moderation Response Format

The response format for content moderation:

A violation item contains the following:

For example:

{
  "output": {
    "violations": [
      {
        "label": "Gambling",
        "start_time": "00:01:14",
        "end_time": "00:01:47",
        "confidence_score": 0.9,
        "relevance_score": 0.9,
        "modality": ["visual", "text on screen", "semantic references"],
        "severity": "low",
        "description": "The video contains visual and text references to gambling platforms, specifically 'Fanduel Sports Network App' and the 'Pechanga' logo, which are associated with sports betting and casino games."
      }
    ]
  },
  "status": "complete",
  "video_id": "[file_id].mp4"
}

​

Content Moderation Complete Example

A complete content moderation example:

import moonshine

moonshine.configure("YOUR-TOKEN-HERE")
index = "my-videos"

items = moonshine.items(index=index)
for item in items:
    file_id = item["file_id"]
    file_name = item["file_name"]
    print(f"Moderating video: {file_name} (ID: {file_id})")

    # Launch the content-moderation flow on each video
    resp = moonshine.run(
        flow="core/content-moderation",
        index=index,
        video_id=file_id
    )

    print(f"Moderation results for {file_name}: {resp}")

​

Content Moderation JSON

TODO