So no, we didn’t have a contingency plan in a glass case labeled "break in event of global pandemic lockdown."
But our founders did design the company and systems not to have a single point of failure, so we wouldn’t be hamstrung by an office or country closing down.
And due to the nature of our company, our unique systems, and careful planning, we’ve managed not to have any operational impact from COVID-19 thus far.
Why? Because it’s not a question of if but when disaster strikes. COVID-19 is unprecedented, but we’ve experienced building fires, seasonal floods, and—in a much happier "disaster"—the Tet holiday shuts down most of our Asian offices for two weeks every year.
We did not make plans for the whole world closing down, but even our worst case scenarios will still leave us with more than 50% of our normal full capacity.
- Janus Matthesen, Pixelz CTO
That means we’re still fulfilling all of our SLAs during the current crisis. We’re delivering on time and on quality (98.77% on-time delivery for March 23-29th), without any service interruption.
That’s because risk mitigation is much more than simply having offices in multiple countries. It’s about flexibility, automation, and being truly geographically independent.
Right now that means the ability to work remotely with adaptable workflows that maintain efficiency and don’t add management overhead. In the next crisis it may mean something quite different. But the key to a resilient system is responsiveness.
In this post I’m going to lay out the reasons why we’re operational, the lessons we learned that you may be able to benefit from, and the virtual community event we’re planning for e-commerce photography leaders to help each other pull through.
The reason we’re still fully operational can be reduced to a one word acronym: S.A.W.™
S.A.W.™ enables automation, responsiveness, and true resilience
Pixelz is powered by S.A.W.™, Specialist Assisted Workflows. It’s the proprietary technology we’ve poured hundreds of thousands of developer hours into over the past decade.
S.A.W.™ is a lean digital retouching assembly line. We break the retouching process down into component steps (a.k.a. “skills”) that are handled by a combination of AI, scripts, and photo editors.
For example, an apparel on-model image might go through 6-10 steps like:
- Retouch: Shape
- Retouch: Garment
- Retouch: Model
- Retouch: Skin Processing
- Layer Mask
- Color Adjustment
- Background Retouch
- Background Extension
- Finalize
Each of those steps is standardized and handled by an auto-assigned person or process.
That’s how we’re able to achieve high levels of consistency at scale, incrementally automate more steps in the retouching process, and increase photo editor satisfaction and retention (they don’t want to do that boring stuff anyway).
And most relevant to this conversation: S.A.W.™ allows us to build incredibly flexible, automated, resilient workflows.
What do I mean by “flexible” and “resilient?”
S.A.W.™ is also our traffic control system. It automatically routes images through steps based on retouching requirements, image deadline, and skill availability.
“Many steps have been automated and are kept intact in an epidemic like this. At the same time, all job assignment and handling between steps is automated, so the human factor is minimized.”
- Quynh Huong, Pixelz Country Manager
There’s no office manager reviewing PDFs and FTP directories and then divvying up batches of images to their colleagues.
Instead, an image comes in and S.A.W.™ sends it to the most optimal step and editor in light of that image’s journey and the impact on all other images in the system. It does the same on completion of that step, then again and again until retouching is complete.
But here’s the most dramatic part—a step might be completed by editors sitting next to each other or in entirely different countries!
An image might go from Hanoi to an AI bot to Dhaka to Da Nang and then final delivery. All without any human assigning it to an editor.
“All locations can back up for each other thanks to the standardization we have in S.A.W.™, so it's easy to switch jobs from one location to the other without much communication.”
- Quynh Huong, Country Manager
Furthermore, while those editors are all in-office during normal times, it’s entirely possible for them to work remotely. They simply need to login to the system, and to S.A.W.™ it’s business as usual.
Because skills are standardized and our workflow is truly geographically independent, we can easily bring more editors online in a new location in event of disaster, or have them work from home in extreme scenarios.
We also have an advanced online training platform (featured in Forbes) that allows us to onboard new editors, train editors on new skills, and refresh editors extremely quickly.
And of course our retouching AI is unaffected by events like COVID-19. All AI operations remain fully intact—and that’s a big deal, because we’ve automated over 50% of our retouching.
Technical infrastructure
“That sounds like a lot of data moving around and loooong wait times,” I hear you say.
So how do we handle bandwidth and up/downloads when we’re moving terabytes of data around the world, and one image to and from editors in different countries?
Over the years, we’ve built 3 levels of caching in addition to AWS’s own. That makes us less reliant on international internet connection speeds.
We have a global cache at a hosting center in Vietnam that also acts as an international hub. That allows us to get our best speeds to and from the office caches at each of our Asian locations.
Each office has its own cache, which is used to get images to editors in-office via their LAN, and to people working from home via their local internet connection. So an editor working in Da Nang, for example, is downloading an image from a cache in Da Nang and not from somewhere further away.
But that’s not all. They’re not sitting there waiting for downloads to complete.
There’s a local cache on the computer the editor is working on. While the editor is working on an image, the computer is downloading and storing their next image (or two) in the background.
“Our goal is to eliminate or at least minimize the time spent waiting for up and download,” says Janus.
Pixelz response to COVID-19
In his message from the Pixelz family on COVID-19, our CEO Thomas Ladefoged said, “our responsibility is to keep our employees, customers, partners, and communities around the world healthy and safe.”
That speaks to the fact that just because we’ve had no operational impact from the Coronavirus doesn’t mean we’ve had no impact. We might be meeting SLAs, but many of us individually are in countries on lockdown, with lives in disarray and health uncertain.
So we’ve taken many measures to try and protect employee health and happiness, as you can read in the message.