July 24th, 2022 Developer Update (Patreon)

Happy Sunday all!

I'm still working on getting the API up and running and in the same update I wanted to tackle something that's been bothering me for a long time: the image pipeline. The image pipeline sucks ass, to say the least. Partly because building it basically went like so:

1. Deploy on-rack compression services and image tools
2. Google says it is not good enough
3. Move to shortpixel because its cheap af
4. Shortpixels API is fucking dogshit
5. All the other services are way too expensive
6. Shortpixel has the best compression, so I guess go back?
7. I have localhost as well, so I guess I should use a temporary s3 bucket for testing purposes?

Now, you don't have to take my word for it. Here are the docs for shortpixels reducer api. While typically I don't badmouth other services this one literally returns the response code in the payload as a string and got me hung up for an hour and a half wondering what the hell was wrong. 

For s3 on localhost, I moved to minio. They offer a localhost version that you can run within Docker and access the s3 API, allowing me to move things around as I see fit. There are still issues, but it works fine. The new pipeline is as follows:

1. Upload image to /pending
2. Kickoff compression job to shortpixel
3. If on localhost, pretend everything was OK and return the same URL in the LossyURL field
4. If production, send it off for processing while still serving all thumbnails\images from pending in the meantime
5. Save the compressed version to the proper location on s3 and update the local images
6. Delay dispatching job to remove orphaned original image(s) to save on space

Other Items In The Update

Along with improvements to the image processing pipeline is better tooling for my setup. I recently moved up into Docker finally and it makes the experience much better for me, but still took a few hours.

Lastly, I have to rewrite the documentation for the API endpoints because, apparently, stoplight yeeted the entire project. If you have a backup download of the openapi or swagger definitions email them to me. Otherwise, gimme a bit to rewrite things. 

Going to try to get this update out next Sunday.