Indie developers are the first to productize state of the art AI models, but incumbents are holding them to high moderation standards — user submitted NSFW/Sensitive prompts get you de-platformed instantly. The worst part? There were no cost effective plug and play moderation solutions available.
Earlier this year, my Co-Founder and I released a popular text-to-video app on the App Store. A small handful of malicious users started creating NFSW content on the platform and we didn’t have the tools to counter them. Google/Apple gave us an ultimatum to fix the issue in 3 days or they would remove our app from their Store.
After a recent review, we found that your app is not compliant with one or more of our Restricted Content policies. Please resolve the issue described below within 3 days of the issuance of this notification or your app will be removed from Google Play.
After looking for solutions online, we realised that there was no cost effective, plug-and-play API available. We tried to implement a solution ourselves, but weren’t able to get something that satisfied Google’s policies in time. As a result, our app was taken down.
Months of hard work, millions of happy users, and tens of thousands of dollars in revenue went down the drain. We were given no chance to appeal or fight back.
After doing some research, we noticed that the issue was widespread. Here's a user on Reddit complaining about OpenAI revoking API access due to malicious users.
If you get banned, you get permanently banned, and they ban you on phone number and email as well.
Got an email warning that something with the API was violating guidelines and that I had 1 week to fix it. The next day I got an email saying I was permanently banned.
They won't tell you what messages violated it, what parts of the guideline you violated
We indie AI developers are at the mercy of the prompts our users choose to input. Moreover, unlike Google/Apple, we don’t have the resources to enable perfect content moderation. This imbalance places an unfair burden on us. We are expected to navigate the same choppy waters as Big Tech but without any manpower.
The reality is that even Google struggles with content moderation. YouTube is full of NSFW content. Yet, paradoxically, they expect near-perfect moderation from us.
In response to this, we developed the Align API. Our aim was twofold: protect the creative freedom of developers while aligning user-generated content with your brand values. We've trained a custom GPT model so you have granular control on what type of content filters through to the end user. We proactively detect NSFW/sensitive topics and reduce costs by preventing malicious prompts getting sent for inference.