New app, new features!

Hello again.

It took a couple weeks, but I've made a huge update to the customization app. This was practically redoing everything from the ground up, at least on the frontend. It is still not perfect, but it is much, much more usable than it was previously.

This ended up being a much longer... and somewhat more meandering... post than I had intended. But a lot has changed! If you're just curious about the specific new features, hop down to that section towards the end

The previous iteration of the app had almost all of the real work being done on the backend in Python. Python is my language of choice, when viable, so it felt natural to me. (I may also irrationally still harbor some old biases against javascript from days back when it was young and immature.) That has quickly proven itself to be the wrong choice. Dealing with the data logistics on the backend is just far more complicated than data that stays on the frontend.

So I moved almost all of the processing to the frontend. I probably shouldn't have been, but I was surprised to learn that there are javascript bindings for OpenCV. The bindings aren't exactly the most polished thing on the web, and they aren't documented particularly well, but they mostly work, which is what really counts. Fixing the backend for this big change was quick and easy. On the one hand it was a little painful to tear down almost all the backend python code I spent weeks developing... on the other hand it was really nice to see the backend greatly, greatly simplified. Almost all of the time I spent working on this over the last couple weeks was dealing with the frontend UI and the image processing pipeline.

The down side to moving processing to the frontend is that, not really being a web developer, I don't have a lot of experience in understanding what actually works well for a wide range of users. I am somewhat concerned that the image processing will just be too much on some devices. I iterated through a couple different processing models over the two weeks, and I ended up landing on a model that attempts to minimize(-ish) memory usage on the device, but it unfortunately does make processing a bit slow, particularly with large images or when certain processing parameters get increased.

To be more specific, in the first iteration (of the revamp) I was storing most of the intermediate images in memory, and when a user setting would change, I was only recalculating the images which were necessary. This had the advantage of being pretty fast, but quite memory heavy. I also didn't feel like it was very clean, with little effective encapsulation of state, it made things very messy, and that was really bogging down my ability to add additional features. The next (current) iteration basically got rid of all of the intermediate image storage. It would just recalculate everything each time. This simplified things quite a bit and made it much easier to extend the processing pipeline with new features. And at first it was almost as quick..! It was only once I had added a whole bunch of new features that the processing started getting slow. But I think that was worth it for the extra features, for now at least. I have some refinements in mind that will hopefully speed things up again by occupying a middle ground, but without introducing a ton of complication.

UI Changes

Completely redone

The UI was completely redone. I think the only piece that remains from the previously released version of the app is the image cropping interface (although there were some changes and bug fixes there, too). The previous iteration was me really just trying to get something working, and I didn't spend a whole lot of time considering mobile browsers (even though that is the vast majority of people these days). Sure enough, some pretty basic things just didn't work or worked really poorly for mobile users. UI going off-screen, now properly responding to touch inputs, etc. The new version was developed mostly approaching it from a mobile point of view. User interfaces are hard to do well, especially when screen space is so limited!

Drawers with processing options

Basically, the processing options were all categorized and placed into drawers in the UI that open to reveal options. I think it works pretty well for what it is. Right now cropping and contrast adjustment are the only things that jump onto a different view to change.

Info buttons

I've included info buttons for pretty much every option that will pop-up explanations for what that option does. Hopefully those are useful. I welcome feedback on the info displayed.

Support for new processing features

A bunch of new processing options were added, mostly related to the idea of foreground/background separation. See the new features section for details.

Full-screen mode

 On the customization app page, you will notice a button near the bottom-left corner of the app. Clicking that will expand the customization app to fill your browser. That is a much more natural way to interact with the app. In full-screen mode a button will appear near the top-left of the screen; press that (or esc on desktop) to pop back out. This never leaves the customization page, it basically just hides the rest of the stuff on the page and locks the app in focus.

As soon as this occurred to me, I realized it was pretty much critical for mobile devices. It makes the customization app immensely more usable.

Using real photos for preview texture

A small change, really, but it was important to me personally. In the previous version of the customization app, the simulated preview of the engraving was using wood textures that were (you may want to sit down for this) generated by AI. That was originally just the quickest way for me to get the preview generation working, and it was good enough.

The new version of the app uses actual photos I took of the stock material I use to produce the engravings. And actually, the frame and the engraved surface are two photos taken of the same piece of stock material. The frame's image texture was just taken after staining the wood.

If it isn't clear, the simulated preview isn't just a photo of the back side of a finished piece. Creating the simulated preview involves several image masks: frame mask, insert mask (I call the piece inside the frame that actually receives the engraving the "insert"), engraving mask. The textures get masked to the area they belong, then composed together to create a preview template (this still happens on the backend). The frontend gets the preview template from the backend, and the engraving mask is used to compose the pipeline output onto the part of the template where it belongs. This gives us a proper orthogonal render of the engraving.

You may also notice the shadow effect and subtle lighting effect on the engraving preview. Those are both effects done on the backend when constructing the template. They really make the simulation look much better!

Would it have been easier to just take a photo of the back of a real product? Definitely. Would have been valid to just take a photo of the back of a real product? Totally. Would it have been smarter to just take a photo of the back of a real product? Very likely. Did it occur to me to just take a photo of the back of a real product? Yes.. but somewhat later than I'd prefer to admit.

But the real reason that I've chosen to do this is composability. The preview I generate looks pretty close to what the photos of actual finished products look like. (The image below is a real photo, for comparison.) But it also solves a future problem, which is what to do when I begin offering engravings of different sizes, shapes, types of wood, etc.

Instead of needing photos for every combination, I can mix and match the wood texture photos with the different masks to simulate the different product variants. And the masks can be generated directly from the CAD designs.

Immediately adds to cart on finalization

I've added a view to the app when the user "finalizes" the customized engraving that shows them their simulated preview one last time with an "Add to cart" button. This view, regardless of the user's screen size, displays the preview at the preview image's natural resolution, and if the image does not fit on screen, allows the user to move it around to see all of it.

In the previous version of the app, when a user finalized, it would add a card to the customization page that had the "Add to cart" button, but that behavior has been absorbed into the app.

(Note: There is still an issue with the site where the shopping cart icon on the upper right does not display a badge indicating the item was added until the page reloads. I haven't had luck fixing that yet, but it's on my list.)

The reason for showing that final view with the Add-to-cart button is an issue that I'm not certain of the best way to handle. It involves the case where a user is on a mobile device in which there is not enough space on their screen for the generated preview to fit using its natural resolution. On the main app view (prior to finalization), the display will scale down the preview to fit on the screen.

This is a (slight) issue because one of the final steps in the processing pipeline is dithering with error diffusion. This process takes a grayscale image and turns it into a binary image, where every pixel is either completely black or completely white. Some day I will make a blog post explaining why this is necessary, but I'll save that for later. The point is (and you can see this if you look closely at any of the examples on this site) the laser engravings themselves are based directly on these dithered images, but when you scale down a dithered image, you lose that binarity, because scaling down involves filtering and interpolation.

When that happens, the image on the user's screen can be qualitatively different than the dithered image. That is why I added the finalization view that forces the image to be displayed at the natural resolution. That guarantees users can see every pixel as it is supposed to be.

Other UI Thoughts

The UI unfortunately does not convey the logic of the processing pipeline terribly well. I'm torn about this, because on the one hand users shouldn't really need to care about specifics like what order the processing stages are done in, but on the other hand, it makes it easier to get good results.

Contrast adjustment is one of the harder settings to deal present to users. Before I ever made this website, I had built processing tools for my own use, but they really rely on understanding the process and what each thing does. Until I made the website I just gave myself raw control over the contrast adjustment parameters. But as soon as I started making an app for other people I realized that there's just no easy way to explain how CLAHE (Contrast Limited Adaptive Histogram Equalization) actually works. So they would be blindly tweaking settings having no idea why they're see what they do.

I had the idea while I was working on the first version of the customization app for the website to simplify this by just giving a handful of preset profiles, showing the results and letting a user pick. That's still where it is now, and the profiles are too limited, but the problem is that the right settings are really specific to the content of the individual image. However, I had an idea this morning for how I might be able to better convey this feature to users as well as giving users the same level of control over it that I have with my local tools. Hopefully I'll be able to get that out in an upcoming update.

New Features

There are lots of new features. Most of them relate to the introduction of the foreground mask. Here is a list of the new features of the image processing.

• Foreground mask generation / split processing
• Foreground mask adjustment
• Background suppression
• Separate foreground/background gamma controls
• Foreground sharpening
• Background blurring
• Modified cropping behavior

Let's get into some of the details.

Foreground Mask Generation / Split-Processing

This was the single most important new feature added to the customization app. With the previous model where most of the processing was done on the backend, it was just too complicated to integrate well.

Now when an image is loaded into the customization app, the app will request ask the backend to estimate a foreground mask of the image. This mask is generated by an AI model running in the backend (model is way too big to run directly in the frontend). This foreground mask is an alpha matte. When it works well it lets us distinguish the foreground from the background areas.

The way to think about that is that the mask contains a number between 0 and 1 for every pixel of the image that essentially gives a confidence level about whether that pixel is part of the background (0) or the foreground(1). If it is 0.5, it is unclear (or, interpreted differently, it is estimating the percentage of that pixels color that comes from the foreground or background. With this interpretation, 0.5 would literally mean it is semi-clear (translucent)).

Having this alpha matte allows us to treat the foreground and background pixels differently in the processing pipeline. That is critical for enhancing the foreground in the engravings.

(Note: This is the only place where AI is used in any of this processing)

Foreground Mask Adjustment

The machine learning model that predicts the foreground mask is certainly not perfect. It works really well on some images and not so well on others. To compensate for this, I've added a couple processing settings that directly operate on the foreground mask itself. Those are erosion/dilation and feathering.

Erosion and dilation are specified as a number of pixels, and they are basically opposites. Erosion will eat away pixels at the edges of the foreground mask (shrinking the foreground region) while dilation will add pixels at the edges, growing the foreground area.

Feathering is also specified as a pixel radius, and it essentially just blurs the edges of the foreground mask. After the foreground and background as split and receive their separate processing, they eventually get merged back together (this is the only place where the fireground mask is actually used), and they get blended together not by just taking a straight average of the pixels in the processed foreground and processed background, but by doing a weighted average according to the foreground mask.

If the edges of the foreground mask are sharp (i.e. the mask values go abruptly from 1.0 to 0.0 at the edge of the predicted foreground), then it is likely that ugly visual artifacts will be produced along those boundaries. By blurring the edges so that the mask fades gracefully from 1.0 to 0.0 at the edges, the foreground and background blend together smoothly.

Background Suppression

Background suppression is a simple switch that basically says "remove the background entirely". If enabled, the background leg of the processing pipeline just treats the background like it is purely white.

When it comes to these engravings, in which a laser burns in the dark areas, white means "no laser power", and so the way to actually think about these grayscale images is not as going between black and white, but actually as going between black and transparent. Because every white pixel of the pipeline's output will simply be the un-burnt wood grain of the wood it is engraved on.

Background suppression is great when all you care about in an image is the foreground figure. The engraving of a Kerbal (from Kerbal Space Program) that I used as the cover image on this post is a good example.

Unfortunately, for images where the estimated foreground mask isn't great, this doesn't necessarily look too good.

Separate Foreground/Background Gamma Controls

There are now separate gamma adjustments for the foreground and the background. This is great for giving the foreground an extra bit of pop. This also is useful in some cases (along with the foreground modification options) where you'd really prefer to suppress the background entirely, but the estimated foreground mask isn't particularly great. You can expand or shrink the foreground mask, feature its edges, and the crank the background gamma way down, and it will smoothly blend away from the foreground figure into nothingness.

Foreground Sharpening

New controls were added for sharpening the foreground image. This involves 2 controls, the radius and amount for sharpening. Sharpening is done using what's called an "unsharp mask" in which a low-pass filtered version of the image is subtracted from the image, resulting in a reduction of low-frequency content. The result is increased contrast around edges in the image. This is really helpful for ensuring that the final engraving doesn't just look like indistinguishable gray mush.

A good engraving isn't really about being perfectly faithful to the brightness of each pixel in the original image. A good engraving will emphasize the boundaries between different areas of semantic content in the image. In a good engraving, you can show someone wearing a dark hoodie in front of a dark-ish background, as long as it's clear where the boundary between them is. Sharpening does exactly that.

The radius parameter is related to the size of the blur in pixels. A small radius will only be able to sharpen on a small scale. A larger radius will be able to sharpen on a wider scale. So ultimately the radius parameter will depend on the size of the features in the image you are tuning.

The amount parameter just scales up the degree of sharpening. If the amount is 0, it won't do anything. If the amount it maxed out, it will probably look terrible regardless of the radius. Use both together to highlight what you want to show!

Background Blurring

This is essentially the opposite of foreground sharpening. Just another way of allowing the foreground to be emphasized, by de-emphasizing distracting background features.

Modified Cropping Behavior

A small but important new feature. The cropping UI now allows you to crop outside of the original image! This is important because this allows you full control over the scale and placement of the subject within the engraving area.

Areas outside the image are filled with white pixels (which makes them behave like they are transparent). The utility of this is limited when not using background suppression, because without it you will likely end up with harsh abrupt edges in the engraving where the edges of the image are.

If you want some background but don't need it to extend to the edge, you can always pre-edit your image in some other software by making sure it fades to white at the edges. That would prevent artifacts from the overcropping from being visible.