October 8th, 2021 Developer Update (Patreon)

Hello Everyone,

I spent some of today preparing a potential move to Cloudflare Images as it provides much better features for us in the long run. If you're not familiar with Image Processing, here is a dive on how it works with MyWaifuList and why it's problematic.

Controlling File Size

At some point this year we moved from attempting to do image compression on rack to doing it via ShortPixel. This not only removes a dependency on image pipeline tools having to be installed on the server, but also frees up operational time. The image are queued on the backend and then a worker simply replaces the uploads that were given to Amazon s3 originally. If it was not, then it would simply take too much time.

Double The Files

Modern image pipelines use serverless functions on Lambda or GCF to handle dynamically serving images in the proper size. The downside is I would need to maintain the functions, make sure to detect headers and serve up the right version, etc to keep google happy. It's a lot of unnecessary pain. 

Right now we are required to pay for:

1. The image upload
2. The image download (to ShortPixel)
3. The image re-upload
4. The thumbnail upload
5. The storage of both the original and the thumbnail
6. Bandwidth out to internet (the major cost)

I was going to originally just deposit everything into a processing bucket and then copy the assets over after processing things with ShortPixel, but that would yield no image at all being shown until the image is finished processing. This would be rather odd when replacing display pictures, etc. 

Cloudflare to the rescue? Maybe?

Cloudflare recently publicly released their Cloudflare Images support. This is a fantastic choice because we can declare variants, upload original files, and simply let Cloudflare handle the rest of the pipeline for us. The pricing is also as follows:

You pay $5/month for every 100,000 stored images and $1 per 100,000 delivered images.

MyWaifuList is currently storing 500,000 files due to having both thumbnails and full versions for nearly everything. AWS is pretty cost effective for long term storage, though, which costs around only 4$ per month in storage fees.

Based on the number of users, series, and waifus we can assume around 300,000 images are stored. That is much more expensive, almost 3x as much just to store the images. The real cost, though, is egress. In Cloudflare, it's not about how much you send but how often you send it. This makes for an interesting issue to solve.

How many requests are we making? Well, that remains to be seen. I have, of today, turned on Cloudwatch request metrics for images to see how many requests we're actually serving up. As long as we're serving less than 200,000 requests per day, then it may make sense.

The threshold is essentially going to be $5-$6 per day as that is what Amazon is costing us, ending up at around 80 - 100 USD per month in fees just to serve up images.

We could also do a partial approach and leave critical items on Cloudflare (waifu images, banners, series images, etc) and serve avatars, etc off of S3 still. We'll see.

Cloudflare R2?

Even if we don't end up getting that, there is room for a mixed approach. For instance, Cloudflare R2 promises to be able to storage objects at about half the cost of S3. However, timed storage is not the issue: bandwidth is. They still say they will be cheaper per operation than others, so we'll have to wait and see.

No Custom Domain (Yet)

If we do move, I'm going to miss seeing images at thicc.mywaifulist.moe. Right now they only offer imagedelivery.net as the base for serving images.

Minor Changes

• Removing perfect scrollbar dependencies, switching to native scrolling with restyling via simplebar 
• Cleaning up old code regions from rushed caching architecture 
• Adding loading wheel to series pages due to azure lane existing. I never imagined a single series would have 400+ waifus.
• Ugly cards on series pages now replaced with new card 
• War panel rebuilt as the old one was simply too slow 
• Merch tab is now removed