<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Pigment's Tech Blog Blog</title>
        <link>https://engineering.pigment.com/</link>
        <description>Pigment's Tech Blog Blog</description>
        <lastBuildDate>Wed, 22 Jul 2026 00:00:00 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <item>
            <title><![CDATA[Choosing to Care as a Leader]]></title>
            <link>https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/</link>
            <guid>https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/</guid>
            <pubDate>Wed, 22 Jul 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[Is there such a thing as female leadership? I've often been asked this question. It may be tempting to reply with the affirmative—to try to « explain » leadership styles, whether our intent is to fit in, include, or antagonize. I'm personally averse to any notion of essentialism, so my own impulse is to scream at this question. As I examine my own path, though, I realize this: female leadership doesn't exist, but the experience of being a woman leader is real. Let's unpack what this means for me.]]></description>
            <content:encoded><![CDATA[<p>Is there such a thing as female leadership? I've often been asked this question. It may be tempting to reply with the affirmative—to try to « explain » leadership styles, whether our intent is to fit in, include, or antagonize. I'm personally averse to any notion of essentialism, so my own impulse is to scream at this question. As I examine my own path, though, I realize this: female leadership doesn't exist, but the experience of being a woman leader is real. Let's unpack what this means for me.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="whats-leadership">What's leadership?<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#whats-leadership" class="hash-link" aria-label="Direct link to What's leadership?" title="Direct link to What's leadership?" translate="no">​</a></h2>
<p>Today, I lead engineering teams at Pigment - joining first as an Individual Contributor, I grew the team from 15 engineers to more than 100 over the last 5 years. My job is to create a vision that engages people around me. If I talk to a developer about « doubling our customer base in the next year », it will not speak to their day-to-day. Making this connection is my job—leveraging business acumen, translating the technology, managing people, organizations and systems. More importantly, my coworkers know who I am, what my values are, how I think, and where we are heading—and why. <strong>This is leadership:</strong> as a leader, you need to bring out the most compelling « why » for your teams.</p>
<p><strong>If we go by this definition, leadership doesn't have a gender.</strong></p>
<p>Yet—more often than not, popular belief associates leadership with traits we could call « masculine ». Taking risks, breaking doors, being loud—traits I would actually consider toxic when they're a permanent mode of leading. It's a widely adopted construct though: <strong>for many, to lead is to be « strong »</strong>. Hence a recurring question: should women leaders adopt these traits? Should women behave like men to be considered acceptable leaders?</p>
<p>Even if that were an answer to the question (it is obviously not), it would be impossible to do. We've all heard this advice: we should « influence without authority ». It sounds reasonable enough, yet it's the most misused advice I've ever heard in tech. Yes, it works pretty well if you are a straight white cis man, say in your 30-40s, and you have to convince other people like you—as is statistically likely to happen in tech. These people may not all be of the same opinion, but they share a profile, thoughts, values—<strong>50% of the decision-making happening in such a group is actually « fit », framed as influence.</strong></p>
<p>But « fit » won't play for women, or any member of an underrepresented group. Stereotypes are commonly associated with the groups we belong to; we're expected to conform to these stereotypes, otherwise, we face pushback. <strong>For women, the required trait is empathy</strong>. I've witnessed this in my career: a coworker being told she was pushy, when she was striving to reach a decision; a promising candidate being rejected for just this reason. I faced hostile or hurt reactions from coworkers when they realized my listening wasn't necessarily agreeing. Further examples are extremely well described in Sheryl Sandberg's book Lean In.</p>
<p>This is a difficult balance to strike, and this may be disheartening: to think that as women leaders, we are forever stuck with being either too hard, or too soft. Yet we exist. We're proof there is another way, and I am convinced this way is action.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="call-to-action">Call to action<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#call-to-action" class="hash-link" aria-label="Direct link to Call to action" title="Direct link to Call to action" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="as-a-leader-regardless-of-your-gender">As a leader (regardless of your gender)<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#as-a-leader-regardless-of-your-gender" class="hash-link" aria-label="Direct link to As a leader (regardless of your gender)" title="Direct link to As a leader (regardless of your gender)" translate="no">​</a></h3>
<p>When I built out the career grid of our engineers at Pigment, the need for seniority levels was challenged by people who perceived them as « titles ». I researched how and why other companies had needed them before us. One reason stood out for me: levels formalize legitimacy against bias, especially when stereotypes will make it harder to recognize our achievements. This need for clarity resonated with my own experience.</p>
<p>Throughout my career, I've often been asked to take on informal responsibilities or to « animate » groups rather than formally managing them. These expectations are disproportionately placed on women, who are often seen as natural facilitators or coordinators. Yet leadership responsibilities should come with the leverage to succeed: clear authority, decision-making power, and organizational support. Responsibilities without leverage are not opportunities, they're setups to fail.
<strong>As a leader, when we trust someone enough to ask more of them, let's stand by our decision and create true leverage.</strong></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="as-a-coworker">As a coworker<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#as-a-coworker" class="hash-link" aria-label="Direct link to As a coworker" title="Direct link to As a coworker" translate="no">​</a></h3>
<p>One day, a coworker told me he didn't see why DEI should concern him. It was genuine—he perceived it as a fight led by groups he didn't belong to. But the question behind inclusivity is broader than this. It asks: are you okay with your coworker being interrupted in meetings, talked over, denied the right to share their ideas and insights? ANY coworker—are you okay with that? What kind of company do you want to work at, and what kind of person do you want to be? <strong>Diversity questions hold up a mirror to us as human beings</strong>.</p>
<p>There is no need to participate in grand projects or to do something absolutely brave—this is a frequent misunderstanding with DEI. <strong>All you have to do is show up</strong>. Make room for other people's opinions, take the time to understand interaction styles, share the stage, literally and figuratively. Beware of passive agreement—one line: « I agree with X and we should do this » moves projects forward when 50 thumbs up emojis cannot.</p>
<p>And if, as a man, you ARE feeling brave: call out exclusive behaviors when you witness them. I've been involved in DEI questions for so long, I regret that so many discussions happen in a closed loop—those who take the strongest stances are already putting themselves on the line every day. The day a man says to another, "What you did is not OK with me, and I do not support it," publicly, he shows he is ready to let go of the relationship and the comfort that comes with it. I am convinced this is when we'll see real progress.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="as-women-leaders">As women leaders<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#as-women-leaders" class="hash-link" aria-label="Direct link to As women leaders" title="Direct link to As women leaders" translate="no">​</a></h3>
<p>I truly believe the greatest mark of privilege is the permission not to care without ever facing consequences. This is why debates about equity get stuck sometimes. Members of an underrepresented group ask for attention from members of a dominant group. This dominant group has the entire freedom to give, or not to give that attention—to believe, or not, facts and figures, no matter how objectively presented. I've seen that script played over and over again. It can be exhausting.</p>
<p>As women leaders, we don't have this privilege. How can we advocate for our own rights while denying them for our neighbors? We can't! <strong>So we care. Not by essence. Because we have to.</strong></p>
<p>And this may be, in fact, our greatest asset as leaders. Company systems reproduce fit—hiring, recognizing, promoting the exact same profiles, over and over. To break that loop and build something different, where diverse opinions are heard, welcomed and valued—we need leaders willing to cross lines. This is where we have leverage. Let's hire and select managers who care, and reflect that in all our decisions, including the difficult ones. Let's recognize behaviors beyond pure technical proficiency. Let's give a push or two behind the scenes to promote equity.</p>
<p>We don't have the power to change everything—I certainly don't. I can't overturn a government's decision to make maths optional in high school (*). I can't create a diverse hiring pipeline out of thin air, without an intentional effort from the rest of the company. I can't halt the current backlash against our rights—at the very best, I try to resist it. But <strong>there is an ecosystem I can build</strong>, values I can share, people I can recruit, support and promote.</p>
<p>For this to happen, I have to be who I am. To accept I lead from my particularities, not against them. <strong>There is no female leadership, only leaders—with the deliberate choice to care.</strong></p>
<small>(*) A measure that had devastating effect on gender parity in scientific studies in France—it has been reverted since</small>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="watch-the-full-discussion">Watch the full discussion<a href="https://engineering.pigment.com/2026/07/22/choosing-to-care-as-a-leader/#watch-the-full-discussion" class="hash-link" aria-label="Direct link to Watch the full discussion" title="Direct link to Watch the full discussion" translate="no">​</a></h2>
<p>Prefer watching? Here's the full discussion (in French, with English subtitles available).</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/c-c_Ok289FA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin"></iframe>]]></content:encoded>
            <category>Leadership</category>
        </item>
        <item>
            <title><![CDATA[Sandboxing LLM-generated code execution]]></title>
            <link>https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/</link>
            <guid>https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/</guid>
            <pubDate>Wed, 10 Jun 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[Pigment provides a central AI platform to organizations for real-time business planning. Pigment AI is based on an agentic architecture that is described in more detail in this blog post.]]></description>
            <content:encoded><![CDATA[<p>Pigment provides a central AI platform to organizations for real-time business planning. Pigment AI is based on an agentic architecture that is described in more detail in <a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/" target="_blank" rel="noopener noreferrer" class="">this blog post</a>.</p>
<p>One of our agents, the Analyst, already had multiple tools to perform simple calculations, such as contribution and variance analysis. In order to add more capabilities, we decided to leverage the code generation feature of LLMs rather than creating a dedicated tool for each capability.</p>
<p>LLM-generated code cannot be trusted by default. It is produced from user-controlled input, which means users may intentionally or accidentally steer the model toward unsafe behavior: reading sensitive data, calling internal services, exfiltrating data, pivoting into internal infrastructure, or exhausting compute resources. From a security perspective, the generated code has to be handled as an untrusted workload.</p>
<p>That requirement led us to build a sandboxed execution environment. In this article, we explain how we went from the initial risk analysis to a proof of concept, and eventually to a production-ready sandbox with support for large datasets.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="constraints">Constraints<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#constraints" class="hash-link" aria-label="Direct link to Constraints" title="Direct link to Constraints" translate="no">​</a></h2>
<p>The code execution solution had to respect the following constraints:</p>
<ul>
<li class="">Execute Python code that has been generated in a previous step of an Agent execution.</li>
<li class="">Allow to pass data to the code to execute</li>
<li class="">The code is executed in a sandboxed environment.</li>
<li class="">The solution must work in the local stack (Docker) as well as in staging and production (Kubernetes).</li>
<li class="">The impact on performance should be minimal.</li>
<li class="">We do not want to introduce a new provider, to avoid having to check it first for price, security, and compliance.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="proving-the-concept">Proving the concept<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#proving-the-concept" class="hash-link" aria-label="Direct link to Proving the concept" title="Direct link to Proving the concept" translate="no">​</a></h2>
<p>At the start of the project, we were not sure that code execution was a viable solution: We needed to provide, with a minimal investment, the Product team with a proof of concept to allow them to check if this approach was relevant for our problems.</p>
<p>For the POC, we chose to rely on the <a href="https://github.com/vndee/llm-sandbox" target="_blank" rel="noopener noreferrer" class="">llm-sandbox library</a>, which supports the two backends we required, Docker and Kubernetes, helps manage Python code execution, including pip package installation, and provides built-in security features such as package restrictions and code vetting. The library also passed our security review for external dependencies.</p>
<p>The main limitation of llm-sandbox was the lack of support for asynchronous operations. This required encapsulating the calls to the library in <code>asyncio.to_thread</code> and handling the session life cycle manually, as in this example:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">@asynccontextmanager</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">async def _get_sandbox_session():</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    # ... [snipped] create the session depending on the backend...</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    await asyncio.to_thread(session.open)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    try:</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">        yield session</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    finally:</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">        await asyncio.to_thread(session.close)</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="security-for-proof-of-concept">Security for proof-of-concept<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#security-for-proof-of-concept" class="hash-link" aria-label="Direct link to Security for proof-of-concept" title="Direct link to Security for proof-of-concept" translate="no">​</a></h3>
<p>The experimentation was done on our staging platform, which provides a production-like environment with a deliberately limited blast radius. It is a completely isolated environment with no production data; its access is restricted to Pigment employees, and breaking it during a deployment or a test is an annoyance, not a disaster.</p>
<p>We relied on two security features of llm-sandbox to implement a security baseline for experimentation in the staging environment.</p>
<p>First, as the library performs the pip package installation in the Python environment in the sandbox, we were able to easily whitelist the packages we needed. In our case, we allowed <code>numpy</code> and <code>pandas</code>.</p>
<p>Second, llm-sandbox provides a regex-based code-vetting feature through security policies: It can generate some language-based regular expressions from some contexts (e.g. detecting module usage) or we can provide it with our own regular expressions to check (for instance, forbidding imports or access to built-ins). The llm-sandbox documentation proposes a handy set of policies to start with. A security policy looks like:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">SecurityPattern(</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    pattern=r"(?m)^\s*(import\s+\w|from\s+\w+\s+import)",</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    description="Import statements are not allowed",</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    severity=SecurityIssueSeverity.HIGH,</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div></code></pre></div></div>
<p>The security policies were advisory: The code had to explicitly call the policy check; each policy could have a description that was returned when a check was not passed. This was quite convenient as we could easily return a message to the agent that called the tool, indicating that the code was unsafe and why. This allowed the LLM to generate new code that fit the restriction.</p>
<p>The specific case of imports is worth detailing. First, imports were prevented by the policies, but we needed to import <code>pandas</code> and <code>numpy</code>. To do so, we simply performed the security checks and we then added a prologue containing the imports. Second, the security checks required the sandbox session to be started, which meant that the container was first created, which took some time and used resources. We quickly noticed that, no matter how the LLM was prompted, it had a tendency to add imports to the generated code and we were wasting time going back and forth between the LLM and the sandbox, creating and deleting a container each time. Adding a simple regex check for imports before creating the containers avoided squandering all these resources.</p>
<p>That said, this code vetting should be treated as a defense-in-depth guardrail, not as the security boundary itself. The problem is inherently difficult: llm-sandbox security policies rely on regex/pattern-based pre-execution checks rather than AST-level analysis, so they cannot reason precisely about every code path, alias, dynamic import, or indirect access pattern. In practice, these rules require continuous hardening as new bypass patterns are identified. For example, allowing a broad library such as pandas may still make lower-level standard-library capabilities reachable indirectly through its internal modules, such as pandas.io.common, which imports modules including os. This may be acceptable for a POC running in a restricted staging environment, but it is far from sufficient as a standalone production security boundary.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="inputs-and-outputs">Inputs and outputs<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#inputs-and-outputs" class="hash-link" aria-label="Direct link to Inputs and outputs" title="Direct link to Inputs and outputs" translate="no">​</a></h3>
<p>Once we had a sandbox with basic security in place, we needed to pass data to this sandbox.</p>
<p>First, we provided a tool to the agent that downloaded the requested data and stored it into a pandas dataframe. This dataframe lived in the execution state of the agent graph and was therefore limited to a few thousand rows of data, which was enough for the POC.</p>
<p>Second, once the sandbox session was created, we were saving the pandas dataframe to a CSV that was then pushed to the sandbox container, thanks to a helper provided by llm-sandbox.</p>
<p>Third, we updated the code prologue (that we added for imports) to load the CSV data into a pandas dataframe. The LLM prompt that was called to generate the code included the name of the variable that contained the pandas dataframe. Later, we added support for multiple data sources (multiple dataframes, multiple CSV files).</p>
<p>Last, the LLM prompt also requested to add a <code>print</code> of the results in the code and to not print anything else. The output of the execution was simply what was displayed in <code>stdout</code>. Here is an example of a generated code (note the lack of imports at the start):</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">df = df1.copy()</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"># Top 3 highest population values across all Country-Year combinations</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">top3 = df.sort_values('Population', ascending=False).head(3).reset_index(drop=True)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">total_population = df['Population'].sum()</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">top3_total = top3['Population'].sum()</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">share_top3 = top3_total / total_population * 100</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">print('Top 3 population values (Country-Year):')</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">print(top3)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">print('\nTotal Population (all rows):', total_population)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">print('Sum of Top 3:', top3_total)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">print('Top 3 share of total:', share_top3)</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="production-ready-sandbox">Production-ready sandbox<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#production-ready-sandbox" class="hash-link" aria-label="Direct link to Production-ready sandbox" title="Direct link to Production-ready sandbox" translate="no">​</a></h2>
<p>The POC was a real success, and it even proved that the upcoming agent release would not be viable without code execution. At that moment, we only had a few weeks to go from a POC to a production-ready solution. The two main missing points were adding in-depth security and finding solutions to handle the expected heavy load.</p>
<p>We considered replacing llm-sandbox with other solutions or approaches (OS-based virtualization, external hosting including sandboxes from LLM providers...). None was giving us the level of control we required.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="in-depth-security">In-depth security<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#in-depth-security" class="hash-link" aria-label="Direct link to In-depth security" title="Direct link to In-depth security" translate="no">​</a></h3>
<p>LLM-generated code could not be treated as trusted, especially because it was derived from user-provided prompts. The regex-based code vetting provided by llm-sandbox was useful as an early rejection mechanism for obviously unsafe or unsupported code, but it was not a complete security boundary. The security boundary must instead be enforced at the infrastructure layer, through runtime isolation, network restrictions, constrained permissions, and dedicated execution environments.</p>
<p>We put some strict restrictions on the workloads that were running in production. Among them:</p>
<ul>
<li class="">Runs on dedicated and isolated workers.</li>
<li class="">Kubernetes logical isolation (namespace).</li>
<li class="">Data isolation between executions: This ensured that we did not risk leaking data between executions.</li>
<li class="">No communication between execution workloads.</li>
<li class="">No network access, be it to the internal network or to the internet.</li>
<li class="">Resistance to local privilege escalation through a restricted pod security standard.</li>
<li class="">Enforcing resource limits.</li>
<li class="">No write access to the root file system.</li>
<li class="">No credentials shall be mounted.</li>
<li class="">Credential-less, permission-less workloads.</li>
<li class="">Minimal, distroless image.</li>
<li class="">Resistance to kernel exploits using <a href="https://github.com/google/gvisor" target="_blank" rel="noopener noreferrer" class="">gVisor</a>.</li>
</ul>
<p>Some of these constraints were added to a pod manifest for the pod that was created from the image. It set limits on CPU and memory, set the root file system as read-only and provided specifically mounted read-write temporary volumes (to store the code and data and have a temp directory). These volumes were RAM disks that disappeared at the end of the execution.</p>
<p>Data isolation between executions was enforced by the lack of communication between running sandboxes but also by terminating the sandbox pod at the end of the execution: A given pod can only be used for a single execution.</p>
<p>Rather than letting llm-sandbox create its own image, we passed our own image (something llm-sandbox directly supports) that implements some of the security constraints mentioned above.</p>
<p>As mentioned, there was a tight deadline, so we sorted the security requirements, balancing the security risk they cover against the implementation costs. This allowed us to progress incrementally and to make sure we would be on time for the general availability of the feature with as many of the most important security features as possible: We chose to cover more straightforward security risks first and to add resistance to the most complex exploits last, as it's a smaller attack surface.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="pre-warming-sandbox-pods-for-low-latency-llm-agent-execution">Pre-warming sandbox pods for low-latency LLM agent execution<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#pre-warming-sandbox-pods-for-low-latency-llm-agent-execution" class="hash-link" aria-label="Direct link to Pre-warming sandbox pods for low-latency LLM agent execution" title="Direct link to Pre-warming sandbox pods for low-latency LLM agent execution" translate="no">​</a></h2>
<p>In the POC, when an LLM agent ran tool calls, we scheduled a fresh isolated pod per execution. This has two impacts on latency: cluster autoscaling (30s+ to provision a node) and image pull on cold nodes (8-12s).</p>
<p>To eliminate this latency for the production release, we set up a placeholder deployment that kept multiple dummy pods permanently running. They looked identical to a real sandbox (same image, same resource reservations). The trick is that each placeholder had a lower scheduling priority than real sandboxes. When a real sandbox had to be started, Kubernetes evicted the low priority dummy pod to make space, which took a few milliseconds. The deployment immediately recreated the evicted placeholder: This was the slow operation, but now performed in the background. The main caveat of this approach is the cost: All the placeholders were actual running pods that are billed by the hosting provider.</p>
<p>The placeholders also kept the sandbox image cached on every node they touched, but a brand-new node had no placeholder yet. An image-warmer DaemonSet closed that gap: an init container pulled the image, a second init container labeled the node with the image version so sandbox pods could require a warm cache via node selector, and a <code>sleep infinity</code> container held the reference so the garbage collector didn't evict it.</p>
<p>With this approach, starting a new sandbox took 1 to 2 seconds on average. This approach only worked if we had enough placeholders in place: We started with a best guess based on the number of expected users and took a wide margin. Setting up monitoring and alerting was key (time to schedule a new sandbox, number of free slots available, etc.), allowing us to tailor the number of placeholder pods to the actual usage.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="supporting-large-datasets">Supporting large datasets<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#supporting-large-datasets" class="hash-link" aria-label="Direct link to Supporting large datasets" title="Direct link to Supporting large datasets" translate="no">​</a></h2>
<p>We actually shipped the first version without support for large datasets: The code execution only had access to a few thousand rows. Surprisingly enough, this covered the majority of use cases, but some of our clients had hundreds of thousands of rows to manipulate, so we decided to add support for them.</p>
<p>The agentic tool (written in Python) was calling another service (written in C#) to fetch the data. We changed the C# service to store the data in Google Cloud Storage instead, and to return an ID to retrieve the data, along with the row count, the column names, and the column types.</p>
<p>We had multiple options to make the data available to the sandbox. Among them:</p>
<ul>
<li class="">Directly fetch data from GCS into the sandbox: This would break the sandbox isolation, giving it access to the internet to reach GCS and, furthermore, giving it access keys to our GCS environments.</li>
<li class="">Download the data at the code execution tool level, then push it to the sandbox: This would download the data to upload it again, which seemed inefficient.</li>
<li class="">Deploy a small in-cluster service that acts as a proxy between sandbox pods and GCS. The sandbox sends a signed URL to the proxy, which validates that the URL points to an allowed GCS bucket, fetches the data, and streams it back: This would be the most secure approach, clearly putting the sensitive operations into the dedicated proxy, unreachable to the LLM-generated code.</li>
</ul>
<p>Downloading then pushing data was the simplest approach, which meant it was at the same time the least expensive and the most secure: The service executing the tools already had GCS credentials, and pushing data to the sandbox was already in place, so we were not adding vulnerabilities. The main concern was around the performance of this operation.</p>
<p>We implemented this approach and performed various measurements.</p>
<p>First, we checked the cost of downloading then uploading data to the sandbox:</p>
<ul>
<li class="">We used a dataset with two columns representing a single dimension metric.</li>
<li class="">A dataset with 990K rows is downloaded then uploaded in less than a second.</li>
<li class="">The download/upload to the sandbox does not vary much with data size: going from a metric with 500K elements to the one with 990K elements adds 0.02 seconds.</li>
</ul>
<p>Then, we measured the impact of the approach on a small dataset (1000 rows):</p>
<ul>
<li class="">The sandbox execution takes about 0.55s longer with the download/upload step.</li>
<li class="">The overall analyst execution takes 0.5s more out of around 25s total, making the download/upload the sole contributor to the slowdown.</li>
</ul>
<p>This performance was perfectly acceptable for a production environment, which proved that this simplest solution was enough.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="conclusion">Conclusion<a href="https://engineering.pigment.com/2026/06/10/sandbox-for-llm-generated-code-execution/#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion" translate="no">​</a></h2>
<p>Going from an idea to a proof of concept to a production-ready sandbox in a few weeks was a demanding challenge, but the incremental approach (prioritizing security risks by impact, shipping the simplest solution that met our constraints, and deferring complexity like large dataset support) proved to be the right strategy. The result is a low-latency, fully isolated code execution environment that has meaningfully expanded the capabilities of the Pigment Analyst Agent.</p>]]></content:encoded>
            <category>AI</category>
            <category>Infrastructure</category>
            <category>Security</category>
            <category>Return of Experience</category>
        </item>
        <item>
            <title><![CDATA[Chasing performance drifts]]></title>
            <link>https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/</link>
            <guid>https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/</guid>
            <pubDate>Thu, 16 Apr 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[You've probably been there. You switched from one tool to another because things got slow. Remember: Webpack, then RsPack or Vite? Jest, then Vitest? Bun? tsgo? Speed is why most of them exist. But performance isn't only a developer problem, our users feel it too.]]></description>
            <content:encoded><![CDATA[<p>You've probably been there. You switched from one tool to another because things got slow. Remember: Webpack, then RsPack or Vite? Jest, then Vitest? Bun? tsgo? Speed is why most of them exist. But performance isn't only a developer problem, our users feel it too.</p>
<p>The trouble with performance is that it drifts slowly. Nobody complains until the day they churn and move to a competitor. At Pigment, performance is in our DNA. We want the platform to stay fast, not just fast at day one.</p>
<p>This article is about how we protect our platform from performance regressions and how we chase performance drifts at Pigment.</p>
<p>We'll walk through three real regressions we caught and fixed. As with many performance regressions, they stayed invisible for a long time until they weren't anymore. For each one, we'll cover both the fix and the trap we set to make sure it could never silently come back.</p>
<div class="theme-admonition theme-admonition-info admonition_xJq3 alert alert--info"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>info</div><div class="admonitionContent_BuS1"><p>Prefer watching over reading? This article is based on a talk — jump to the <a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#watch-the-talk" class="">video</a> at the end, it covers the same content.</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="performance-for-everyone">Performance for everyone<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#performance-for-everyone" class="hash-link" aria-label="Direct link to Performance for everyone" title="Direct link to Performance for everyone" translate="no">​</a></h2>
<p>Our users care about fast tools just as much as we do. They prefer a snappy interface over a sluggish one. But they don't measure bundle size or API response times, they experience your platform as a black box. They don't care if the backend is slow, as long as we can make the experience feel fast.</p>
<p>Many companies have measured the cost of slowness on their bottom line. In 2006, Google found that adding half a second to each search dropped traffic by 20% (<a href="https://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html" target="_blank" rel="noopener noreferrer" class="">More details here</a>). Amazon found that 100ms of extra page load time cost them 1% in sales (<a href="https://assets.website-files.com/61060433cb5cbb34f58da08c/61065835a1cb346f7673c592_StanfordDataMiningAmazonCaseStudy.pdf" target="_blank" rel="noopener noreferrer" class="">More details here</a>).</p>
<p>Performance clearly matters. But maintaining good performance is hard, for two reasons.</p>
<p>First, performance issues are everywhere. A <a href="https://stackinsight.dev/blog/memory-leak-empirical-study/" target="_blank" rel="noopener noreferrer" class="">recent study</a> across 500 repositories found that 86% of them had memory leaks and by extension performance problems.</p>
<p>Second, it drifts silently. As engineers, do you still manually test against every browser? Against the slow machine most of your users actually have? On a slow network? When we build, we tend to forget that we're on powerful machines with fast networks and optimal setups. We rarely test against real production data as we often use trimmed-down datasets rather than ones that mimic the craziest real-world cases. Some regressions stay invisible to us. And it doesn't take a sweeping refactor to introduce one: a single line of CSS, a misplaced hook, a missing cleanup can silently flip a well-optimized experience into a sluggish one.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="defining-performance-scope">Defining performance scope<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#defining-performance-scope" class="hash-link" aria-label="Direct link to Defining performance scope" title="Direct link to Defining performance scope" translate="no">​</a></h2>
<p>At Pigment, we are building a Business Planning platform. Part of that is visualizing data in what we call Boards: a single place mixing Grids, Charts and KPIs. Boards are also fully real-time: if a colleague updates data you're currently viewing, it triggers a partial reload for you.</p>
<p>From day one, Boards have been critical. This is where our customers spend their time, investigate and dig into their data. It cannot be slow so we always pay close attention to anything touching it. That said, even with careful attention, regressions crept in. That's what this article is about.</p>
<p><img decoding="async" loading="lazy" alt="An example of a Board" src="https://engineering.pigment.com/assets/images/hello-pigment-8e52a36a59240e69f168e45f4ae7204c.png" width="1862" height="1314" class="img_ev3q"></p>
<p>Before diving in: from the very start, our Boards have been optimized with virtualization for Grids and with smart partial reloads to avoid unnecessary recomputation of charts.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="out-of-memory">Out of memory<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#out-of-memory" class="hash-link" aria-label="Direct link to Out of memory" title="Direct link to Out of memory" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-report">The report<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#the-report" class="hash-link" aria-label="Direct link to The report" title="Direct link to The report" translate="no">​</a></h3>
<p>It happened. A client reported that our UI was suddenly crashing. At the time, most usage was around 1,000 cells in the Grids. We dug in, tried to reproduce it... and couldn't. So we went back to the customer asking for more details. They reported an out-of-memory error and told us how they used the platform in their day-to-day: they opened a Pigment tab at the start of the week, left it running alongside many other tabs and came back to it later; by then, it had crashed.</p>
<p>That user flow gave us a crucial clue. Everything in Pigment is real-time: whether you're actively using it or not, it reloads itself whenever needed. At that time, we weren't cutting real-time updates even on hidden, unused tabs. But while that aggravated the problem, it wasn't the root cause. The real issue was that every render leaked a tiny slice of memory. We were registering event listeners and failing to clean them up, which caused leaks that pulled along surrounding values until the tab ran out of memory. We were leaking states of all our cells plus the entire DOM of a cell. With 1,000 cells, that adds up fast.</p>
<p>The problematic snippet was something like:</p>
<div class="language-jsx codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-jsx codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">useEffect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> </span><span class="token function-variable function" style="color:#d73a49">callback</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">addEventListener</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"keydown"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> callback</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// ⬇️ listener</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword control-flow" style="color:#00009f">return</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">removeEventListener</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"keyup"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> callback</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// ❌ wrong event type</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">[</span><span class="token punctuation" style="color:#393A34">]</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<p>Leaks are everywhere and most of the time invisible until they blow up your application.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="setting-the-trap">Setting the trap<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#setting-the-trap" class="hash-link" aria-label="Direct link to Setting the trap" title="Direct link to Setting the trap" translate="no">​</a></h3>
<p>Finding and fixing the bug was one thing. Stopping there would have left the door open for the same class of regression to creep back in. So we looked for ways to prevent it.</p>
<p>A few options surfaced. Some were local: <em>check that the event listener is properly removed by mocking handlers in a unit test</em>. Others were more general: <em>find a lint rule to catch these patterns</em>. But none of them were enough. Leaking is easy and no tooling can catch every variant at the static analysis level. We needed something that could surface leaks on the actual user flow.</p>
<p>During our investigation, we found that the leak was retaining state from unmounted components. So we decided to track whether state from critical components was leaking. We came up with the following test flow:</p>
<ul>
<li class="">Open the app on the homepage</li>
<li class="">Count the number of probed states</li>
<li class="">Run a flow</li>
<li class="">Go back to the homepage</li>
<li class="">Count the number of probed states</li>
</ul>
<p>In our case, the "Run a flow" was replaced by "Open a Board".</p>
<p>To track this, we instrumented key components. The instrumentation is only active in test context. In production, we replaced it by a no-op. All it takes is calling a new hook from our key components:</p>
<div class="language-jsx codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-jsx codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function maybe-class-name" style="color:#d73a49">ComponentName</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">props</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">useLeakProber</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// no-op in Production</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic">//...</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>Under the hood, the hook stores a <code>WeakRef</code> to a piece of component state. Unlike a regular reference, a <code>WeakRef</code> doesn't prevent the garbage collector from reclaiming the object. So if all strong references to the state get removed, it gets collected and <code>ref.deref()</code> returns <code>undefined</code>. In case of a leak, the state stays alive and <code>deref()</code> still returns the object. That's the signal we count.</p>
<p>Here is a simplified version of our <code>useLeakProber</code>:</p>
<div class="language-jsx codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-jsx codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> </span><span class="token literal-property property" style="color:#36acaa">probes</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token maybe-class-name">WeakRef</span><span class="token tag punctuation" style="color:#393A34">&lt;</span><span class="token tag" style="color:#00009f">object</span><span class="token tag punctuation" style="color:#393A34">&gt;</span><span class="token plain-text">[] = [];</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain-text"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain-text">function useLeakProber() </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">[</span><span class="token plain">probe</span><span class="token punctuation" style="color:#393A34">]</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">useState</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">{</span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">useEffect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    probes</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">push</span><span class="token punctuation" style="color:#393A34">(</span><span class="token keyword" style="color:#00009f">new</span><span class="token plain"> </span><span class="token class-name">WeakRef</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">probe</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">[</span><span class="token plain">probe</span><span class="token punctuation" style="color:#393A34">]</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain-text"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain-text"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain-text">window.countActiveLeaks = function countActiveLeaks() </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword control-flow" style="color:#00009f">return</span><span class="token plain"> probes</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">filter</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">ref</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> ref</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">deref</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">!==</span><span class="token plain"> </span><span class="token keyword nil" style="color:#00009f">undefined</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">length</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>With the helper in place, we could start using it. Our take was that neither unit nor integration tests were enough for memory leaks. They can catch specific cases we've already identified, but they're too narrow to surface unknown leaks. So we went for end-to-end tests. Cypress was the de facto choice in the industry at that time and it was ours too. The test looked like this:</p>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">describe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"No leak"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">it</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"should not leak"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">login</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">visitHome</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">countLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">then</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">leakCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">flow</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">visitHome</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">expectAtMostLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">leakCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">countLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">gc</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword control-flow" style="color:#00009f">return</span><span class="token plain"> cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">window</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">then</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">gc</span><span class="token operator" style="color:#393A34">?.</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token keyword control-flow" style="color:#00009f">return</span><span class="token plain"> </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">countActiveLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">expectAtMostLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">leakCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">gc</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">window</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">should</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">gc</span><span class="token operator" style="color:#393A34">?.</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> activeLeaks </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">countActiveLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">activeLeaks</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">to</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">be</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">at</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">most</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">leakCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>One subtle Cypress detail: <code>.should</code> retries the predicate until it either succeeds or times out. That's why we trigger the garbage collector inside it on every retry. This matters when dealing with memory: browsers prefer to grow the heap until they hit a limit rather than eagerly free it. Without explicitly triggering GC, leaked objects can hang around for hours even after they're no longer needed. Additionally calling GC once isn't enough as it may just flag memory as "to be freed later." The trick is to keep retrying until the count reaches our expectations. If it never does, you're still holding state in memory.</p>
<p>Adapted to our Boards, the test was simply:</p>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">describe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"No leak"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">it</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"should not leak when browsing a Board"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">login</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">visitHome</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">countLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">then</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">c</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">visitBoard</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"board-xyz"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">visitHome</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">      </span><span class="token function" style="color:#d73a49">expectAtMostLeaks</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">c</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<div class="theme-admonition theme-admonition-caution admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>caution</div><div class="admonitionContent_BuS1"><p>We tried it on a toy project and the probes stayed silent, even against a deliberate leak. It works in our stack, likely because we have a mix of React version, third-party libs and patterns like Suspense or lazy that create the right conditions for it. That was good enough for us to stop there. Worth trying in yours, but don't assume silence means no leak.</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="sluggish-keystrokes">Sluggish keystrokes<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#sluggish-keystrokes" class="hash-link" aria-label="Direct link to Sluggish keystrokes" title="Direct link to Sluggish keystrokes" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-report-1">The report<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#the-report-1" class="hash-link" aria-label="Direct link to The report" title="Direct link to The report" translate="no">​</a></h3>
<p>This ticket came up later. By then, clients had started using Pigment with much larger datasets. We had gone from Grids of 1,000 cells to Grids with 1,000,000. At that scale, small performance issues become very visible.</p>
<p>The report was simple: keyboard navigation felt slow. We loaded the client's data and reproduced it immediately. Part of why we hadn't caught it earlier: we'd always tested with developer-crafted, trimmed datasets and never against real data for many reasons including legal concerns.</p>
<p>By toggling on the React devtools we quickly spotted something that looked problematic: we were re-rendering all the cells anytime we changed the focused cell.</p>
<p><img decoding="async" loading="lazy" alt="Re-render drama" src="https://engineering.pigment.com/assets/images/re-render-drama-98a65256e319d1c2b111a134344fbdb7.gif" width="1541" height="462" class="img_ev3q"></p>
<p>The fix itself was straightforward once we understood the cause. The selection state lived on the Grid component which wraps all cells so that every cell could read it. But by design, when a React component's state changes, it re-renders itself and all its descendants. We were triggering that on every keystroke, touching thousands of cells at once. In React 16, that's a fully synchronous operation that blocks the main thread until the entire subtree is traversed. Newer React with concurrent rendering would have avoided the main-thread block, but the display would still have felt laggy.</p>
<p>The fix: move the selection state outside of React to a mechanism that React doesn't control.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="setting-the-trap-1">Setting the trap<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#setting-the-trap-1" class="hash-link" aria-label="Direct link to Setting the trap" title="Direct link to Setting the trap" translate="no">​</a></h3>
<p>As with the first issue, we wanted to make sure this couldn't quietly come back. The specific trigger was user selection, but the pattern is general: any feature that tracks per-cell state faces the same risk. Our strategy:</p>
<ul>
<li class="">Count the number of renders</li>
<li class="">Run a flow</li>
<li class="">Count the number of renders</li>
</ul>
<p>Same as before, we added a new hook, called from key components:</p>
<div class="language-jsx codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-jsx codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> renderCount </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">new</span><span class="token plain"> </span><span class="token class-name">Map</span><span class="token operator" style="color:#393A34">&lt;</span><span class="token plain">string</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> number</span><span class="token operator" style="color:#393A34">&gt;</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">useRenderCount</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter literal-property property" style="color:#36acaa">kind</span><span class="token parameter operator" style="color:#393A34">:</span><span class="token parameter"> string</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">useEffect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    renderCount</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">set</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">kind</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">renderCount</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">get</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">kind</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">??</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">+</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">1</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>With the hook in place we were able to shape a generic implementation for re-render tests:</p>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">describe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"No unwanted re-render"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">it</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"should not re-render"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">login</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">visitPage</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">resetRenderCounters</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">flow</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expectRenderCount</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"kind-a"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> countA</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token comment" style="color:#999988;font-style:italic">// And maybe others: expectRenderCount('kind-b', countB);</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">resetRenderCounters</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">window</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">then</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">renderCount</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">clear</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">expectRenderCount</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">kind</span><span class="token parameter punctuation" style="color:#393A34">,</span><span class="token parameter"> count</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">window</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">should</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> observedCount </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">renderCount</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">get</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">kind</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">observedCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">to</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">be</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">eq</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">count</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>Applied to our specific case, the test was:</p>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">describe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"No unwanted re-render on keyboard navigation"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">it</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"should not re-render all grid when moving between cells"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">login</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">visitGrid</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"grid-name"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">focusOnCell</span><span class="token punctuation" style="color:#393A34">(</span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">resetRenderCounters</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">pressArrowDown</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expectRenderCount</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"cell"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">2</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expectRenderCount</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"header"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="slowdown-on-scroll">Slowdown on scroll<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#slowdown-on-scroll" class="hash-link" aria-label="Direct link to Slowdown on scroll" title="Direct link to Slowdown on scroll" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-report-2">The report<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#the-report-2" class="hash-link" aria-label="Direct link to The report" title="Direct link to The report" translate="no">​</a></h3>
<p>Scale grew again. From millions of cells, we moved to billions. From one or two items per enum cells, we moved to millions. Not everything fit in memory anymore, so we introduced lazy loading and started to fetch only the data at display time for large datasets.</p>
<p>A client reported something interesting: the more they scrolled their Grid, the slower it got. Investigating, we traced it to our data cache. Our algorithm inserted new entries one at a time with a linear scan. While scrolling, we weren't inserting one entry but many. The result: a quadratic slowdown that got worse and worse the more data was already loaded.</p>
<p>The fix was to batch the insertions.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="setting-the-trap-2">Setting the trap<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#setting-the-trap-2" class="hash-link" aria-label="Direct link to Setting the trap" title="Direct link to Setting the trap" translate="no">​</a></h3>
<p>Once again, fixing it wasn't enough. We wanted to make sure it couldn't come back. So we built a dedicated test suite.</p>
<p>The concept: detect long tasks. A long task is any synchronous operation that blocks the browser's main thread for more than 50ms. Past that threshold, the browser can't process scroll events, keyboard input or animations, so users experience it as a freeze. Browsers expose long tasks through the <code>PerformanceObserver</code> API. There's simply no reason to hit one while scrolling a large dataset. So we built a test that sets up a large dataset in CI and fails if any long task is detected.</p>
<p>The detection was done using this snippet:</p>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> longTasks </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">[</span><span class="token punctuation" style="color:#393A34">]</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> observer </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">new</span><span class="token plain"> </span><span class="token class-name">PerformanceObserver</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter">list</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword control-flow" style="color:#00009f">for</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> entry </span><span class="token keyword" style="color:#00009f">of</span><span class="token plain"> list</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">getEntriesByType</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"longtask"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    longTasks</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">push</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">entry</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">duration</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">observer</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">observe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"> </span><span class="token literal-property property" style="color:#36acaa">entryTypes</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">[</span><span class="token string" style="color:#e3116c">"longtask"</span><span class="token punctuation" style="color:#393A34">]</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<p>We quickly realized there was no reason to limit this check to the big-dataset test. Every test in our suite should be free of long tasks. So we made it a global <code>afterEach</code>:</p>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token function" style="color:#d73a49">describe</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"Any test"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">afterEach</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expectNoLongTask</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic">// The test</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">function</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">expectNoLongTask</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">window</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">should</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">(</span><span class="token parameter dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token arrow operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> observedCount </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token dom variable" style="color:#36acaa">window</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">longTasks</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">length</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token function" style="color:#d73a49">expect</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">observedCount</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">to</span><span class="token punctuation" style="color:#393A34">.</span><span class="token property-access">be</span><span class="token punctuation" style="color:#393A34">.</span><span class="token method function property-access" style="color:#d73a49">eq</span><span class="token punctuation" style="color:#393A34">(</span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="takeaways">Takeaways<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#takeaways" class="hash-link" aria-label="Direct link to Takeaways" title="Direct link to Takeaways" translate="no">​</a></h2>
<p>Chasing performance drifts is hard. We decided to complement reviews, unit tests, integration tests and linting with an extra layer: end-to-end tests tailored to performance. At Pigment, this layer is the last line of defense before a regression reaches production. Ideally we catch things earlier but each of the three patterns above has proven useful by catching regressions that slipped through everything else.</p>
<p>The underlying strategy is always the same:</p>
<ul>
<li class="">Measure something</li>
<li class="">Run a flow</li>
<li class="">Measure something again</li>
</ul>
<p>In the first case, the measure was a memory proxy. In the second, a render count. In the third, the number of long tasks. This pattern won't catch everything as there are always new proxies to add. But it gives a simple and reusable safety net that's easy to reason about and extend.</p>
<p>If this is the kind of challenge that grabs your interest, we should talk! We're always on the lookout for talented engineers to join our teams.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="watch-the-talk">Watch the talk<a href="https://engineering.pigment.com/2026/04/16/chasing-performance-drifts/#watch-the-talk" class="hash-link" aria-label="Direct link to Watch the talk" title="Direct link to Watch the talk" translate="no">​</a></h2>
<p>This article is a written version of a talk we gave at the React Paris conference in 2026.</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/7BbALMqYDCA?si=T-472D-rWLnF2Th1" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin"></iframe>]]></content:encoded>
            <category>Performance</category>
            <category>Front</category>
        </item>
        <item>
            <title><![CDATA[Impersonation Done Right: Tokens, Read-Only Guarantees, and Audit Trail]]></title>
            <link>https://engineering.pigment.com/2026/04/08/safe-user-impersonation/</link>
            <guid>https://engineering.pigment.com/2026/04/08/safe-user-impersonation/</guid>
            <pubDate>Wed, 08 Apr 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[If you build enterprise SaaS with any kind of permissions model, you've heard this feature request before: "Our admins need to see what a user sees." The motivation is always the same — a user reports they can't access something, or they're seeing the wrong data, and the admin has no good way to verify what's going on from the user's perspective.]]></description>
            <content:encoded><![CDATA[<p>If you build enterprise SaaS with any kind of permissions model, you've heard this feature request before: "Our admins need to see what a user sees." The motivation is always the same — a user reports they can't access something, or they're seeing the wrong data, and the admin has no good way to verify what's going on from the user's perspective.</p>
<p>The workarounds your customers resort to are painful. They ask their users for screenshots, or schedule screen shares, but these are high-friction interactions. They try to reproduce the issue with test accounts, but test accounts never have the same permissions, the same data, or the same organizational context as the real user.</p>
<p>So sooner or later, the feature request lands on your backlog: let admins log into the app <em>as another user</em> — see the same dashboards, the same data, the same permissions. Even before users complain about missing access.</p>
<p>But the moment you say "admin impersonation" in an enterprise context, alarm bells go off. Can the admin modify the user's data? Can they access private information? Is there an audit trail? How do you prevent this from becoming a backdoor?</p>
<p>These are fair concerns. At Pigment, we built a <strong>user-facing impersonation</strong> mode that lets admins review what users see safely. It's been in production for a while now, and our customers love it. Here's how we designed it.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="design-principles-we-started-with">Design principles we started with<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#design-principles-we-started-with" class="hash-link" aria-label="Direct link to Design principles we started with" title="Direct link to Design principles we started with" translate="no">​</a></h2>
<p>Before writing any code, we agreed on four non-negotiable principles that would guide every design decision:</p>
<p><strong>Read-only by default.</strong> Impersonation is for <em>observing</em>, not <em>acting</em>. An admin investigating a user's dashboard should never accidentally create, update, or delete anything. If the impersonation feature can't guarantee this, it shouldn't exist.</p>
<p><strong>Time-bounded sessions.</strong> No indefinite impersonation. A session should last long enough to investigate an issue — 30 minutes — and then automatically expire. If the admin needs more time, they can start a new session. This limits the blast radius of a compromised or forgotten session.</p>
<p><strong>Full audit trail.</strong> Every impersonation session is logged: who impersonated whom, when, in which organization. This is non-negotiable for compliance (SOC 2, GDPR) and for building trust with customers. The impersonated user's organization can see these logs.</p>
<p><strong>Scoped to the impersonator's own access.</strong> Impersonation should never grant access to parts of the platform the impersonator can't already reach. If the impersonated user belongs to multiple organizations but the impersonator only has permission in one of them, they can only see what the user sees in that one organization. The impersonation session is the intersection of both users' access, not a union.</p>
<p>That said, the precise definition of "the impersonator's own access" matters here. We don't restrict based on what the impersonator currently has access to — we restrict based on what they <em>could</em> access, given their role. Impersonation is restricted to Security Admins, who by definition control the access configuration for every user in the organization. A Security Admin who doesn't currently have access to a given resource could grant themselves that access at any moment.</p>
<p>Because of this, during an impersonation, we present a faithful, complete picture of what that user sees — not a degraded version filtered through the admin's own current permissions. The trust model rests on the fact that this capability is restricted to users who already hold organizational-wide power over access configuration.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-token-architecture">The token architecture<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#the-token-architecture" class="hash-link" aria-label="Direct link to The token architecture" title="Direct link to The token architecture" translate="no">​</a></h2>
<p>When an admin starts impersonating a user, the backend doesn't just flip a flag on the existing session. Instead, it issues a <strong>brand-new JWT token</strong> with a specific set of claims that encode everything the system needs to know:</p>
<ul>
<li class=""><strong>The impersonated user's identity</strong> — so every downstream service sees the world as that user. Same permissions, same data access, same organizational context.</li>
<li class=""><strong>The impersonator's identity</strong> — preserved in a separate claim, so the audit trail always knows who's really behind the request. This claim travels through every service in the request pipeline.</li>
<li class=""><strong>A read-only flag</strong> — embedded directly in the token, not inferred from context. More on this in the next section.</li>
<li class=""><strong>An expiration time</strong> — 30 minutes from issuance. After that, the token is simply invalid.</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Endpoint tagging diagram" src="https://engineering.pigment.com/assets/images/diagram-endpoing-tagging-7fbdd34ad6f73a2e40ad25a2189915fe.png" width="1552" height="1081" class="img_ev3q"></p>
<p>This approach has a key advantage: every backend service sees the impersonated user's identity as the primary identity. We don't need to teach each service about impersonation — they just see a normal user request. The impersonation metadata rides alongside, invisible to services that don't need it, available to those that do (like audit logging).</p>
<p>On the <strong>frontend</strong>, the impersonation token is stored separately from the admin's regular authentication token. The admin's own session is never lost or overwritten. When impersonation ends — either because the admin stops it or the token expires — the frontend simply reverts to the original token. The transition is seamless: one moment you're seeing the app as the user, the next you're back to your own view.</p>
<p>This separation also means the frontend can display a clear visual indicator during impersonation. The admin always knows they're in an impersonation session, reducing the risk of confusion. It also enables per-tab impersonations: you can work side by side, immediately seeing the result of a configuration change.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="read-only-enforcement">Read-only enforcement<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#read-only-enforcement" class="hash-link" aria-label="Direct link to Read-only enforcement" title="Direct link to Read-only enforcement" translate="no">​</a></h2>
<p>The read-only guarantee is the heart of the trust model, so let's dive into how we enforce it.</p>
<p>An important design goal is that the admin sees the app <em>exactly</em> as the user sees it — including edit buttons and action menus. This is the whole point: the admin needs to review what the user can and cannot do. The UI is unchanged during impersonation. But if the admin clicks an edit button, the backend rejects the mutation and the frontend displays a clear message explaining that writes are blocked during impersonation.</p>
<p>The read-only flag lives in the JWT itself — it's not a frontend-only guard. Every API request passes through an authentication middleware that checks this flag. If the request targets a mutating endpoint and the session is read-only, the middleware rejects it with a 403 and a specific response header before it ever reaches any business logic. The frontend recognizes this header and surfaces a user-friendly error.</p>
<p>A key concern for us was <strong>maintainability</strong>. We introduce new API endpoints every day across many teams, and the question that kept coming up during design was: "How do we make sure this doesn't silently break six months from now?" The read-only enforcement lives in middleware, but the real insurance policy is a custom build-time check that verifies every single endpoint is explicitly tagged as either allowed or denied during read-only sessions. If a developer adds a new endpoint without this annotation, the build fails. (For those familiar with .NET: it's a Roslyn analyzer.)</p>
<p>I'll be honest: when we first discussed writing a custom linter rule for this, there was some debate whether it was worth the effort. But in practice, it has been one of the best investments we made. It forces an explicit decision for every endpoint — no silent defaults that could mask a missed review. Months after the initial implementation, with dozens of new endpoints added by teams that had nothing to do with the impersonation feature, the invariant still holds.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-road-not-taken-infrastructure-level-enforcement">The road not taken: infrastructure-level enforcement<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#the-road-not-taken-infrastructure-level-enforcement" class="hash-link" aria-label="Direct link to The road not taken: infrastructure-level enforcement" title="Direct link to The road not taken: infrastructure-level enforcement" translate="no">​</a></h2>
<p>Before settling on endpoint tagging, we seriously explored a fundamentally different approach: enforcing read-only at the infrastructure level rather than the API level. We built a functional proof-of-concept early on, and the idea was appealing in its elegance.</p>
<p><img decoding="async" loading="lazy" alt="Infrastructure-level enforcement diagram" src="https://engineering.pigment.com/assets/images/diagram-infra-enforced-c531e68cc66154a189ffb738d50f60bf.png" width="1550" height="1081" class="img_ev3q"></p>
<p><strong>Treat the business logic as a black box.</strong> Instead of deciding which endpoints are read-only, intercept all outgoing calls to infrastructure services — Postgres, ElasticSearch, GCP, email, ... — and block writes at that level. For Postgres specifically, the approach is to maintain two separate connection pools against the same database — one with standard read-write connections, one where every connection is opened with <code>SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY</code> — and route every request in a read-only session exclusively through the latter. If any code path attempts a write through the read-only pool, Postgres itself throws an error with a specific error code. Catch that exception, bubble it up, return a custom header to the frontend.</p>
<p><strong>Why this was tempting:</strong> it's truly generic. Business logic becomes a black box. Any new code path that writes to the database is automatically blocked, with zero effort from feature teams. No annotation to remember, no build-time check needed. The approach scales effortlessly as the codebase grows.</p>
<p><strong>Why we discarded it:</strong> the real world is messier than theory.</p>
<p>First, context propagation across service boundaries proved complex. Our architecture involves inter-service communication through RabbitMQ and gRPC. Propagating the read-only context through message headers and call metadata added significant plumbing — and every new communication channel would need the same treatment.</p>
<p>Second — and this was the killer — not all writes during a read-only session are illegitimate. Revoking an authentication token writes to the database. Computing statistics when rendering a view updates cache entries. Audit logging itself writes to the database. The infrastructure-level approach can't distinguish between "this write is the user's action" and "this write is a side-effect of reading." Each legitimate write needs an override mechanism, and once you start identifying which writes to allow, you're doing essentially the same work as tagging endpoints — except at a lower level of abstraction with less context about intent.</p>
<p>Third, every infrastructure client needs its own read-only implementation. Postgres has native support for read-only sessions, but not every service does — and whenever we integrate a new one, someone needs to remember to implement read-only enforcement for it. No linter rule can catch a missing infrastructure-level guard the way our build-time check catches a missing endpoint annotation.</p>
<p><strong>What we ended up with</strong> is what we initially considered the fallback: endpoint tagging with a build-time linter. It turned out to be more pragmatic — simpler to implement, easier to reason about, and the build-time check addresses the maintainability concern that made infrastructure-level enforcement attractive in the first place.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="audit-logging">Audit logging<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#audit-logging" class="hash-link" aria-label="Direct link to Audit logging" title="Direct link to Audit logging" translate="no">​</a></h2>
<p>Every impersonation session start is recorded in the audit log with the impersonated user's identity — their ID, email, and display name. This log is scoped to the organization, meaning the organization's compliance team can review it.</p>
<p>But the audit trail goes deeper than just "session started." Because the impersonator's identity is embedded in the JWT and travels through every service in the request pipeline, any action taken during an impersonation session can be attributed back to the impersonator. If something looks suspicious in the logs, you can always answer the question: "who was actually behind this request?"</p>
<p>This transparency is a feature, not just a compliance checkbox. When customers know that every admin action is tracked and visible, they trust the system more. It turns impersonation from a scary backdoor into a legitimate, accountable support tool.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="making-it-pleasant-to-use">Making it pleasant to use<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#making-it-pleasant-to-use" class="hash-link" aria-label="Direct link to Making it pleasant to use" title="Direct link to Making it pleasant to use" translate="no">​</a></h2>
<p>Getting the security right is necessary, but not sufficient. If the impersonation experience is clunky, admins won't use it — and you're back to screenshots and screen shares.</p>
<p>We built an <strong>impersonation widget</strong> in the sidebar that shows who you're currently impersonating and lets you end the session with one click. A subtle dark border frames the entire screen so the admin always knows they're in an impersonation session. And critically, you can <strong>switch directly</strong> from impersonating one user to another without stopping the session first. When an admin is investigating a permissions issue across multiple users, this makes the workflow fluid rather than tedious.</p>
<p>We also customized the <strong>"not found" page</strong> during impersonation. If the impersonated user doesn't have access to the page the admin was looking at, instead of a generic "page not found," we show a specific message: "The impersonated Member does not have access to this page." This is valuable information — often it's exactly what the admin was trying to find out. The error page includes a button to stop impersonation and return to the admin's own view.</p>
<p>And a small touch we're proud of: the UI is displayed in the <strong>impersonator's locale</strong>, not the impersonated user's. You're seeing the user's data and permissions, but the interface still speaks your language.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="beyond-user-impersonation-support-sessions-and-nesting">Beyond user impersonation: support sessions and nesting<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#beyond-user-impersonation-support-sessions-and-nesting" class="hash-link" aria-label="Direct link to Beyond user impersonation: support sessions and nesting" title="Direct link to Beyond user impersonation: support sessions and nesting" translate="no">​</a></h2>
<p>So far we've described user impersonation — an admin within a customer's organization reviewing what one of their users sees. But we actually support multiple kinds of impersonation, and they compose.</p>
<p><strong>Support sessions</strong> are used by Pigment employees to access a customer's environment for troubleshooting. Unlike user impersonation, support sessions are <em>not necessarily</em> read-only — a support engineer may need to fix configuration issues or run diagnostics. These sessions have their own safeguards: they require multi-factor authentication, they're logged through a separate security pipeline, and they're scoped to specific organizations.</p>
<p>The interesting part is that these two modes <strong>nest</strong>. A Pigment support engineer can start a support session, and then from within that session, start a user impersonation to see the app as a specific customer user. This is a common workflow: "Customer X reports that User Y can't see their dashboard — let me check."</p>
<p>If you're thinking "nested impersonation sounds terrifying," that's a reasonable reaction. But the safeguards compose as well. The nested session inherits the stricter constraints: user impersonation is always read-only, regardless of whether the outer support session allows writes. The full chain of identities is preserved in the token — the original support engineer, the service account used, and the impersonated user — so the audit trail remains complete. And the organization scoping still applies: the support account must have access to the organization in which the impersonated user is being impersonated.</p>
<p>Crucially, the customer remains in full control: if they don't want Pigment support to impersonate their users, all they have to do is remove the impersonation permission from the support account in their organization. No ticket, no negotiation — it's a standard permission they manage themselves.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-this-project-taught-me">What this project taught me<a href="https://engineering.pigment.com/2026/04/08/safe-user-impersonation/#what-this-project-taught-me" class="hash-link" aria-label="Direct link to What this project taught me" title="Direct link to What this project taught me" translate="no">​</a></h2>
<p>Impersonation is one of those features that's easy to build naively and hard to build safely — and even harder to maintain long-term. If I had to distill it to one insight: get the token architecture right. Separate tokens, embedded constraints, dual identities — the rest follows naturally from there.</p>
<p>But beyond the technical design, this project reminded me why I enjoy working at Pigment. We explored an ambitious infrastructure-level approach — proof-of-concept branches, deep trade-off debates — because it deserved to be explored. And when the evidence pointed toward a simpler solution, we pivoted. That balance between ambition and pragmatism is something I value deeply.</p>
<p>I was also struck by the level of thought our designers put into the experience. The dark border, the inline impersonation switcher, the custom "not found" page, the locale handling — none of these were afterthoughts. They came from designers who cared about getting the details right. The result feels polished, not just safe.</p>
<p>When customers first started asking for this, I had this exciting feeling we could build it safely — but turning that intuition into a production feature took a whole team of talented people, much thought, and many iterations. Seeing it become something our customers genuinely rely on is one of the most rewarding experiences I've had as an engineer.</p>]]></content:encoded>
            <category>Security</category>
            <category>Software Design</category>
        </item>
        <item>
            <title><![CDATA[Being an Engineering Program Manager at Pigment — Podcast (in French)]]></title>
            <link>https://engineering.pigment.com/2025/09/08/being-an-engineering-program-manager-at-pigment-podcast/</link>
            <guid>https://engineering.pigment.com/2025/09/08/being-an-engineering-program-manager-at-pigment-podcast/</guid>
            <pubDate>Mon, 08 Sep 2025 00:00:00 GMT</pubDate>
            <description><![CDATA[We’re delighted to announce that Mélissa Djuimo and Matthieu Allegre, two of our Engineering Program Managers (EPM), were invited to the Tech Lead Corner podcast to share their experience and discuss the specific EPM role at Pigment⁠.]]></description>
            <content:encoded><![CDATA[<p>We’re delighted to announce that Mélissa Djuimo and Matthieu Allegre, two of our Engineering Program Managers (EPM), were invited to the Tech Lead Corner podcast to share their experience and discuss the specific EPM role at Pigment⁠.</p>
<p><a href="https://podcast.ausha.co/tech-lead-corner-3/melissa-djuimo-et-matthieu-allegre-etre-epm-chez-pigment" target="_blank" rel="noopener noreferrer" class="">🎧 Tech Lead Corner | Mélissa Djuimo and Matthieu Allègre - Being EPM at Pigment</a>.</p>
<p>Mélissa Djuimo and Matthieu Allegre are pillars of our EPM team. Their expertise covers FinOps, Enablement community, and R&amp;D applications for Mélissa, and Performance program as well as Compute &amp; Storage community for Matthieu⁠.
⁠</p>
<h1>Why This Podcast?</h1>
<p>This invitation is part of our commitment to sharing experiences and increasing visibility around Pigment's innovative practices in technical organization. Since the EPM role is relatively specific and still uncommon, it was important for us to contribute to the tech community by explaining how we've structured this role and how it integrates into our organization⁠.
⁠</p>
<h1>Topics Covered</h1>
<p>During this French-language episode, our EPMs discussed:</p>
<ul>
<li class="">The evolution of Pigment's tech organization since its founding in 2019</li>
<li class="">The emergence of the EPM role and the problems it addresses</li>
<li class="">Their personal journeys to this profession</li>
<li class="">Concrete examples of cross-functional initiatives</li>
<li class="">Interaction with technical teams</li>
<li class="">The qualities required to be a good EPM at Pigment</li>
</ul>
<h1>Listen to the Episode</h1>
<p>The podcast is available on the Ausha platform, and we encourage you to listen to it to better understand this key function in our organization⁠ — <a href="https://podcast.ausha.co/tech-lead-corner-3/melissa-djuimo-et-matthieu-allegre-etre-epm-chez-pigment" target="_blank" rel="noopener noreferrer" class="">🎧 Listen to the Podcast</a>.</p>
<p>A big thank you to Mélissa and Matthieu for this excellent representation of Pigment and our approach to Engineering Program Management!</p>]]></content:encoded>
            <category>Podcast</category>
        </item>
        <item>
            <title><![CDATA[How We Turbocharged Our Prediction Pipeline Using Dask]]></title>
            <link>https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/</link>
            <guid>https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/</guid>
            <pubDate>Tue, 02 Sep 2025 00:00:00 GMT</pubDate>
            <description><![CDATA[At Pigment, we provide a central platform for organizations to model and plan using their most valuable data. As part of that mission, enabling predictive insights is key. Our first implementation of predictions worked well for smaller datasets, but as more users began applying the feature to larger and more complex data models, we quickly hit the limits of a single-machine setup in terms of memory and compute.]]></description>
            <content:encoded><![CDATA[<p>At Pigment, we provide a central platform for organizations to model and plan using their most valuable data. As part of that mission, enabling predictive insights is key. Our first implementation of predictions worked well for smaller datasets, but as more users began applying the feature to larger and more complex data models, we quickly hit the limits of a single-machine setup in terms of memory and compute.</p>
<p>To address this, we redesigned our infrastructure around a distributed Dask cluster, allowing us to scale horizontally and support much larger workloads. In this post, we’ll share the architecture of our new solution and key takeaways from adopting Dask in production.</p>
<p><img decoding="async" loading="lazy" alt="First version of the prediction pipeline" src="https://engineering.pigment.com/assets/images/first-version-7c52296065d9213b66017a1c1e1c3fec.png" width="1659" height="699" class="img_ev3q"></p>
<p>In our initial design, a prediction request was initiated from the UI and handled entirely by two backend pods (one C#, one Python). They would:</p>
<ol>
<li class="">Load all the necessary data from the database,</li>
<li class="">Forward that data via gRPC to another pod running the forecasting model in Python,</li>
<li class="">Wait for the forecast results, and then return them to the frontend.</li>
</ol>
<p>While this setup was a good fit for a MVP, it quickly revealed several limitations as soon as users began applying forecasts to larger datasets. The key issues were:</p>
<ul>
<li class="">Fragile Long-Running Execution: Forecasting is a time-intensive process. If any gRPC or HTTP call failed—for example, due to a frontend reload or a pod restart—the entire prediction job would be interrupted and lost.</li>
<li class="">Memory Bottlenecks: All data needed to be loaded into memory on a single backend pod. If the dataset was too large to fit, the pod would crash with an out-of-memory (OOM) error.</li>
<li class="">CPU Constraints: The actual forecast computation, handled by the Python model pods, was CPU-bound. This limited throughput and made the system difficult to scale efficiently for larger or concurrent predictions.</li>
</ul>
<h1>Improved V2 Architecture</h1>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="queue-based-communication">Queue-Based Communication<a href="https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/#queue-based-communication" class="hash-link" aria-label="Direct link to Queue-Based Communication" title="Direct link to Queue-Based Communication" translate="no">​</a></h3>
<p>To improve reliability and resilience, we transitioned from gRPC-based communication to a queue-based architecture. This shift ensures that if no processing pod is available at a given moment, incoming requests are simply queued and processed as soon as resources become available. Additionally, if a pod is interrupted or shuts down mid-task, it can safely requeue its message, allowing another pod to resume processing without loss.</p>
<p>Since queues don’t support streaming in the same way gRPC does, we changed the data flow to use GCS (Google Cloud Storage) as an intermediate store. Specifically, data to be predicted is exported as Avro files to GCS and later consumed by Python-based prediction models. The model results are also written back to GCS in the same format.</p>
<p>The updated architecture is illustrated below:</p>
<p><img decoding="async" loading="lazy" alt="Queue-based architecture" src="https://engineering.pigment.com/assets/images/queue-based-communication-6e05b2c7ab091336583990afa7dba5a2.png" width="2001" height="1916" class="img_ev3q"></p>
<ul>
<li class="">Step 1: Export: The Predict Backend pushes an export request to the Export queue. This request is picked up by a dedicated system (Export Backend), which reads data from the database and writes it as an Avro file on GCS. Once the export is completed, a separate (not shown) notification queue signals the Predict Backend that the file is ready.</li>
<li class="">Step 2: Predict: Upon receiving the export completion signal, the Predict Backend pushes a prediction request to the Predict queue. The Predict Models service picks this up, reads the GCS file, performs the prediction, and writes the output back to GCS as another Avro file. It then sends a notification (via another queue not shown) to inform the Predict Backend that predictions are ready.</li>
<li class="">Step 3: Import: The Predict Backend now triggers the import phase by pushing a message to the Import queue. The Import Backend imports the predictions into the database as a new Metric. Finally, another queue notifies the Predict Backend once the import is completed. After this flow, the Predict Backend sends a “Prediction Done” event to the UI, enabling the user to view the forecasted values.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="implementing-a-dask-cluster">Implementing a Dask Cluster<a href="https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/#implementing-a-dask-cluster" class="hash-link" aria-label="Direct link to Implementing a Dask Cluster" title="Direct link to Implementing a Dask Cluster" translate="no">​</a></h3>
<p>At this stage, the Predict backend still ran as a single Python pod. This limited our ability to process large GCS files that couldn’t fit in memory and prevented us from distributing CPU load across multiple pods, which in turn made jobs slower.</p>
<p>To address this, we evaluated Apache Spark and Dask as parallel computing frameworks. We chose Dask because it integrates seamlessly with NumPy, preserves our existing Pythonic code and mental model, supports parallelism and out-of-core (bigger-than-memory) computation, and avoids the JVM/SQL-heavy shift required by Spark. Additionally, Spark comes with a steeper learning curve for our use case.</p>
<p>We refactored our workload to be Dask-compatible (a more complex task than initially expected) and introduced a Dask scheduler and Dask workers into our infrastructure. Instead of executing tasks directly, the Python pod now submits them to the scheduler, which distributes work across the workers.</p>
<p>Dask also provides Kubernetes-native operators, making it straightforward to integrate these components into our cluster. For example, the built-in DaskAutoScaler queries the scheduler about the workload and dynamically scales worker pods up or down to match demand.</p>
<p>Finally, the execution pattern was updated: each worker reads a partition of the input GCS file and writes back its corresponding partition of the output file. A key factor that enables this parallelization is the fact that our prediction models work on time series independently - each time series can be forecasted without knowledge of other series. This constrains us in terms of prediction models, but it also allows us to distribute the workload efficiently across workers, resulting in better resource utilization, and the ability to handle datasets much larger than a single pod's memory capacity.</p>
<p><img decoding="async" loading="lazy" alt="Dask cluster" src="https://engineering.pigment.com/assets/images/dask-cluster-a4adb22e3e57ba8c6d1d75f75ab25422.png" width="2175" height="979" class="img_ev3q"></p>
<h1>Challenges</h1>
<p>During implementation we faced several significant challenges.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="blue-green-deployment">Blue-Green Deployment<a href="https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/#blue-green-deployment" class="hash-link" aria-label="Direct link to Blue-Green Deployment" title="Direct link to Blue-Green Deployment" translate="no">​</a></h3>
<p>A critical requirement was ensuring that the Python library versions remained perfectly synchronized across the client, the Dask scheduler, and all Dask workers. Any mismatch would trigger warnings and could result in subtle, hard-to-debug errors.</p>
<p>To enforce this guarantee, we adapted our ArgoCD deployment pipeline to follow a blue-green deployment strategy. Using kustomizations and replacements, the commit SHA1 is appended to Kubernetes resources, ensuring that client-1 communicates only with scheduler-1. When deploying client-2, we can be confident that it only targets scheduler-2. This approach prevents cross-version communication between components during rollout.</p>
<p><img decoding="async" loading="lazy" alt="Blue-green deployment" src="https://engineering.pigment.com/assets/images/blue-green-deployment-c173e9bf06f02dcaad9a4027a5da8a1d.png" width="1841" height="679" class="img_ev3q"></p>
<p>Another difficulty emerged with the Dask Autoscaler. While it successfully scaled worker pods up in response to workload spikes, the additional pods would remain alive indefinitely, even after the cluster became idle. In extreme cases, a single job could spawn over 200 workers, creating a substantial cost burden if they were never released.</p>
<p>To address this, we dug into the autoscaler’s source code and ultimately submitted a pull request to the Dask repository. Although the fix hasn’t been merged yet, we currently run a custom version of the library that incorporates our patch.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="missing-features">Missing Features<a href="https://engineering.pigment.com/2025/09/02/how-we-turbocharged-our-prediction-pipeline-using-dask/#missing-features" class="hash-link" aria-label="Direct link to Missing Features" title="Direct link to Missing Features" translate="no">​</a></h3>
<p>Some required features were not available out-of-the-box, which forced us to build custom solutions:</p>
<ul>
<li class="">
<p>Graceful Worker Shutdowns:</p>
<p>By default, workers could be terminated even if they were still processing tasks or had queued work. This sometimes caused tasks to be dropped permanently, leaving jobs stuck. To resolve this, we developed our own SchedulerPlugin that ensures active or queued tasks finish before a worker is killed.</p>
</li>
<li class="">
<p>Prediction Job Cancellation:</p>
<p>While Pigment’s UI allows cancellation of a prediction job, simply cancelling the associated Dask Future was insufficient — subtasks spawned by that job would continue running. To fix this, we implemented a custom mechanism:</p>
<ul>
<li class="">Each task is annotated with the Prediction Job ID.</li>
<li class="">When a job is cancelled, all tasks carrying the same ID are terminated.</li>
<li class="">This process is executed in a loop, since even dying tasks can spawn new subtasks that also need to be cleaned up.</li>
</ul>
</li>
</ul>
<p>The final result is a more robust and cost-efficient deployment pipeline, better autoscaling behavior, and safer job handling tailored to Pigment’s needs.</p>
<h1>Conclusion</h1>
<p>Adopting Dask was not without its challenges, but the benefits have been transformative. By moving from a single-pod execution model to a fully distributed cluster, we unlocked the ability to process massive datasets reliably, scale resources dynamically, and deliver faster predictions to our customers.</p>
<p>This shift has made our prediction pipeline far more resilient, cost-efficient, and capable of handling workloads that were previously out of reach. Most importantly, it enables Pigment to provide predictive insights at the scale our customers need, empowering them to make better, data-driven decisions with confidence.</p>
<p>Here is a glimpse of Dask’s mesmerizing UI when running a prediction job.</p>
<p><img decoding="async" loading="lazy" alt="Dask Job" src="https://engineering.pigment.com/assets/images/dask-job-dbedab6ddb98db287d4b6957e2a7df23.avif" class="img_ev3q"></p>]]></content:encoded>
            <category>Software Design</category>
            <category>Infrastructure</category>
            <category>Performance</category>
        </item>
        <item>
            <title><![CDATA[Agentic AI for data analysis in Pigment]]></title>
            <link>https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/</link>
            <guid>https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/</guid>
            <pubDate>Thu, 31 Oct 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Agentic AI, in which one or multiple AI agents can pursue goals autonomously or collaboratively, represents a new frontier in generative AI. Unlike traditional large language models (LLMs) that operate independently, these systems combine specialized agents with unique roles to handle complex, open-ended tasks. Here, we’ll explore how Pigment uses this approach for data analysis, empowering clients to gain deeper insights.]]></description>
            <content:encoded><![CDATA[<p>Agentic AI, in which one or multiple AI agents can pursue goals autonomously or collaboratively, represents a new frontier in generative AI. Unlike traditional large language models (LLMs) that operate independently, these systems combine specialized agents with unique roles to handle complex, open-ended tasks. Here, we’ll explore how Pigment uses this approach for data analysis, empowering clients to gain deeper insights.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="pigment-insights-feature">Pigment "Insights" feature<a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/#pigment-insights-feature" class="hash-link" aria-label="Direct link to Pigment &quot;Insights&quot; feature" title="Direct link to Pigment &quot;Insights&quot; feature" translate="no">​</a></h2>
<p>At Pigment, we’re developing an AI assistant that performs in-depth analysis on client data, identifying patterns and extracting business insights that would otherwise be difficult to uncover. A typical dataset may consist of a <em>metric</em> (e.g., sales) defined across several <em>dimensions</em> (e.g., region, quarter, product). Common tasks include explaining metric values (e.g., "What contributed to the total cost in 2024?") or investigating differences between figures (e.g., "Why are Q3 sales lower than last year?").</p>
<p>While tools like <a href="https://github.com/Sinaptik-AI/pandas-ai" target="_blank" rel="noopener noreferrer" class="">PandasAI</a> can perform AI-assisted data analysis on a given dataset, our implementation within Pigment’s SaaS app is unique in several ways:</p>
<ul>
<li class=""><strong>Targeted Data Access</strong>: Rather than pulling in the entire dataset (which is often very large), the AI assistant retrieves specific data projections from Pigment’s backend, focusing only on relevant dimensions.</li>
<li class=""><strong>Interactive Visualizations</strong>: We leverage Pigment’s interactive <em>views</em> (diagrams showing data), providing users with a consistent experience across AI and non-AI features. This consistency removes the need for the assistant to generate visualizations through code.</li>
</ul>
<p>From the point of view of the Pigment backend, the Insights assistant acts as an independent module, communicating with users in natural language, retrieving data as needed, and generating <em>view structures</em> that are then populated with data by the backend to create final visualizations.</p>
<p><img decoding="async" loading="lazy" alt="User Flow" src="https://engineering.pigment.com/assets/images/user-flow-d9da8627fa7e333bdf77d41ff403f90c.png" width="2016" height="1316" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="traditional-llm-chaining-vs-agentic-ai">Traditional LLM chaining vs agentic AI<a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/#traditional-llm-chaining-vs-agentic-ai" class="hash-link" aria-label="Direct link to Traditional LLM chaining vs agentic AI" title="Direct link to Traditional LLM chaining vs agentic AI" translate="no">​</a></h2>
<p>LLM chaining, a more traditional approach for handling complex tasks, relies on a sequence of steps where each subsequent step follows based on predefined rules and previous outputs. At Pigment, we use LLM chaining for certain structured tasks, such as producing visuals that answer a question, like "Show me actual sales vs. budget."</p>
<p>However, for nuanced data analysis (often initiated by an open-ended "why" question), the path to insight is less clear. A classic example is <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox#Kidney_stone_treatment" target="_blank" rel="noopener noreferrer" class="">Simpson's paradox</a>, where aggregating data may lead to seemingly contradictory results. Addressing these challenges often calls for a divide-and-conquer strategy — a natural fit for agentic AI, as each agent can tackle segmented tasks independently and efficiently.</p>
<p>Recent advancements in LLMs, including <a href="https://openai.com/o1/" target="_blank" rel="noopener noreferrer" class="">OpenAI's o1 model</a>, are beginning to address more complex logical tasks. Agentic AI extends these capabilities by positioning LLMs as the “brains” of agents that can use specialized <em>tools</em> (such as data retrieval functions or external sources) and collaboratively solve problems. Here, multiple agents work together with back-and-forth interactions, often requiring a supervisory agent to orchestrate these exchanges. This design allows agentic systems using lightweight LLMs to achieve results comparable to, and sometimes surpassing, more advanced non-agent LLMs.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="agent-design-with-langgraph">Agent design with LangGraph<a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/#agent-design-with-langgraph" class="hash-link" aria-label="Direct link to Agent design with LangGraph" title="Direct link to Agent design with LangGraph" translate="no">​</a></h2>
<p>For agent design, we use <a href="https://langchain-ai.github.io/langgraph/" target="_blank" rel="noopener noreferrer" class="">LangGraph</a> framework, which integrates seamlessly with LangChain and allows for detailed customization of agent interactions. In LangGraph, the agent system is represented as a graph where each agent and tool acts as a node, with specific interaction rules forming the connections.</p>
<p><img decoding="async" loading="lazy" alt="Graph Diagram" src="https://engineering.pigment.com/assets/images/graph-diagram-1e66f222607b52576b0bf5709843a004.png" width="1275" height="597" class="img_ev3q"></p>
<p>In our setup:</p>
<ul>
<li class="">The <code>DataAnalyst</code> agent calls the underlying LLM and manages data requests and filter manipulations via its tools.</li>
<li class="">A human-in-the-loop is incorporated via a special <code>FinalResponse</code> node. Even though it's not an executed tool, it is called via the <em>tool calling</em> interface to ensure the output is structured and contains the summary and the suggested next steps.</li>
<li class="">Currently, a <code>Supervisor</code> agent ensures each process concludes with a user response, and in the future, it will manage more complex analysis flows and agent handoffs.</li>
</ul>
<p>A typical agent conversation might look like:</p>
<ul>
<li class=""><strong>Human</strong>: "Can you help me best understand contributions to the total amount of Product Sales (7430)?"</li>
<li class=""><strong>DataAnalyst</strong>: calls <code>BreakdownDimension</code> tool<!-- -->
<ul>
<li class=""><strong>BreakdownDimension</strong> tool: "Breakdown over dimension Quarter: ..."</li>
<li class=""><strong>BreakdownDimension</strong> tool: "Breakdown over dimension Region: ..."</li>
</ul>
</li>
<li class=""><strong>DataAnalyst</strong>: "Here's what I found so far ..."</li>
<li class=""><strong>Supervisor</strong>: "Please call <code>FinalResponse</code> function to finish the conversation."</li>
<li class=""><strong>DataAnalyst</strong>: calls <code>FinalResponse</code> with a summary: "Q4 24 is the strongest quarter, contributing the highest sales...", next steps: "Focus on the top contributing regions: EUROPE, UK", "Investigate the strong performance in Q4 24"</li>
<li class="">(next turn) <strong>Human</strong>: "Focus on Q4 24"</li>
<li class=""><strong>DataAnalyst</strong>: calls <code>SelectItems</code> tool</li>
<li class="">etc</li>
</ul>
<p>The core architecture of the agent graph is outlined in the following Python code:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token comment" style="color:#999988;font-style:italic"># Initialize a new State graph (the most common type of graph in LangGraph)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> StateGraph</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">AgentState</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token comment" style="color:#999988;font-style:italic"># Define graph nodes</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> _build_data_analyst_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">llm_model</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> executed_tools</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> final_response_tool</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"Tools"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> _build_tools_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">executed_tools</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"Supervisor"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> _build_supervisor_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"FinalResponse"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> _build_final_response_node</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">set_entry_point</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token comment" style="color:#999988;font-style:italic"># Define conditional edges from the `DataAnalyst` node</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_conditional_edges</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">	source</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">	path</span><span class="token operator" style="color:#393A34">=</span><span class="token plain">_route_from_data_analyst</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">	path_map</span><span class="token operator" style="color:#393A34">=</span><span class="token punctuation" style="color:#393A34">{</span><span class="token string" style="color:#e3116c">"Tools"</span><span class="token punctuation" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"Tools"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"FinalResponse"</span><span class="token punctuation" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"FinalResponse"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"Supervisor"</span><span class="token punctuation" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"Supervisor"</span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token comment" style="color:#999988;font-style:italic"># Define non-conditional edges that route back to the `DataAnalyst` node</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_edge</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">start_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"Tools"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> end_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_edge</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">start_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"FinalResponse"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> end_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token plain">add_edge</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">start_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"Supervisor"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> end_key</span><span class="token operator" style="color:#393A34">=</span><span class="token string" style="color:#e3116c">"DataAnalyst"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token comment" style="color:#999988;font-style:italic"># Compile graph and add a breakpoint before the `FinalResponse` node - we will manually inject the user input to the state and relaunch the graph.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">compiled_graph </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> graph</span><span class="token punctuation" style="color:#393A34">.</span><span class="token builtin">compile</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">interrupt_before</span><span class="token operator" style="color:#393A34">=</span><span class="token punctuation" style="color:#393A34">[</span><span class="token string" style="color:#e3116c">"FinalResponse"</span><span class="token punctuation" style="color:#393A34">]</span><span class="token punctuation" style="color:#393A34">)</span><br></div></code></pre></div></div>
<p>The <code>_build_node</code> functions contain the internal logic of each node, while the <code>_route_from_data_analyst</code> function manages the conditional switching between nodes.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="moving-toward-multi-agent-systems">Moving toward multi-agent systems<a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/#moving-toward-multi-agent-systems" class="hash-link" aria-label="Direct link to Moving toward multi-agent systems" title="Direct link to Moving toward multi-agent systems" translate="no">​</a></h2>
<p>Agentic AI is particularly effective when tasks can be divided, with specialized agents handling each part. This division of labor prevents overloading an LLM with complex prompts. In Pigment, data often involves interconnected metrics linked by <em>formulas</em> written in Pigment’s language. A <code>FormulaAnalyst</code> agent could assist in analyzing input metrics' impact by retrieving relevant documentation and applying specialized tools. <code>DataAnalyst</code> and <code>FormulaAnalyst</code> can work independently on their tasks, reporting to the <code>Supervisor</code> node that becomes the graph entry point. This modular approach allows for future agents to handle tasks like outlier detection or time-series analysis.</p>
<p><img decoding="async" loading="lazy" alt="Graph Diagram With Formulas" src="https://engineering.pigment.com/assets/images/graph-diagram-with-formulas-d419e0ff38718f039f8b3c6f1456590c.png" width="1275" height="657" class="img_ev3q"></p>
<p>Here is a conversation addressing the <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox#Kidney_stone_treatment" target="_blank" rel="noopener noreferrer" class="">Simpson's paradox</a> (comparing two treatments for kidney stones):</p>
<ul>
<li class=""><strong>Human</strong>: "Can you help me best understand why the success rate is higher for Treatment B?"</li>
<li class=""><strong>Supervisor</strong>: calls <code>FormulaAnalyst</code></li>
<li class=""><strong>FormulaAnalyst</strong> (after calling its tools): "The formula calculates the success rate by dividing the summed <code>Succeeded</code> metric by the summed <code>Trials</code> metric, after removing the <code>Stone_size</code> dimension from both metrics."</li>
<li class=""><strong>Supervisor</strong>: calls <code>DataAnalyst</code></li>
<li class=""><strong>DataAnalyst</strong> (after calling its tools): "Considering the above, the observed higher overall success rate for Treamtent B is due to a disproportionate number of trials with small stones, which have a generally higher success rate, creating a bias when comparing treatments".</li>
<li class=""><strong>Supervisor</strong>: calls <code>FinalResponse</code> summarizing the findings.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="takeaways">Takeaways<a href="https://engineering.pigment.com/2024/10/31/agentic-ai-for-data-analysis/#takeaways" class="hash-link" aria-label="Direct link to Takeaways" title="Direct link to Takeaways" translate="no">​</a></h2>
<p>Our experience with agentic AI has been promising, especially for handling complex data analysis tasks. However, this approach presents both advantages and challenges:</p>
<ul>
<li class=""><strong>creativity vs consistency</strong>: Generative AI enhances creativity, enabling novel insights and perspectives, though it occasionally brings an element of unpredictability. Using agents rather than standalone LLMs provides added flexibility, but it may also increase unpredictability. At Pigment, our backend safeguards data integrity in views, and we are considering adding a verification agent to further reinforce accuracy in generated text.</li>
<li class=""><strong>scope management</strong>: The assistant suggests next steps, which sometimes extend beyond its intended scope. During lengthy conversations, it may recommend broader business optimizations, which we consider a <em>bug rather than a feature</em> (we plan a separate Pigment assistant for business knowledge questions). System prompt adjustments help refine this scope but require continuous fine-tuning.</li>
<li class=""><strong>debugging and testing</strong>: Testing agentic systems can be complex due to multiple agents and tools. Automated testing is challenging as responses aren’t always strictly correct or incorrect, though we can test for expected elements like view generation and analysis flow.</li>
</ul>
<p>Overall, agentic AI provides the adaptability needed for detailed data analysis and shows significant potential as we continue to add new agents and functionalities.</p>]]></content:encoded>
            <category>AI</category>
            <category>Return of Experience</category>
        </item>
        <item>
            <title><![CDATA[Image Snapshot testing with Cypress]]></title>
            <link>https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/</link>
            <guid>https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/</guid>
            <pubDate>Mon, 07 Oct 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Unit tests, integration tests, end-to-end tests — like many modern development teams, at Pigment, we rely heavily on these methods to minimize the risk of functional regressions in our codebase. In this post, we'll explore how we added image snapshots to our Cypress end-to-end tests as an extra safeguard for catching visual regressions.]]></description>
            <content:encoded><![CDATA[<p>Unit tests, integration tests, end-to-end tests — like many modern development teams, at Pigment, we rely heavily on these methods to minimize the risk of functional regressions in our codebase. In this post, we'll explore how we added image snapshots to our Cypress end-to-end tests as an extra safeguard for catching visual regressions.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-need">The need<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#the-need" class="hash-link" aria-label="Direct link to The need" title="Direct link to The need" translate="no">​</a></h2>
<p>As our front-end application grew in size and complexity, we started noticing a lack of sufficient coverage against visual regressions. While our design system components are well covered with <a href="https://jestjs.io/docs/snapshot-testing" target="_blank" rel="noopener noreferrer" class="">Jest snapshots</a>, which stores markup and CSS for later comparison, we felt there was a gap when it came to ensuring visual stability at the dashboard level. To give you a clearer picture of what I mean, below is an example of a Pigment dashboard, straight from our corporate website:</p>
<figure><p><img decoding="async" loading="lazy" alt="Pigment dashboard" src="https://engineering.pigment.com/assets/images/pigment-dot-com-dashboard-fb0ea8e29354095b5073278fb2a75b2f.png" width="1316" height="846" class="img_ev3q">
</p><figcaption><em>Part of a Pigment dashboard copied from our website</em></figcaption><p></p></figure>
<p>So while individual component tests — including Jest HTML + CSS snapshots — cover a lot of ground, we were increasingly aware that for more complex views like dashboards, we'd need an additional layer of visual testing to safeguard against regressions.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="enter-image-snapshots">Enter image snapshots<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#enter-image-snapshots" class="hash-link" aria-label="Direct link to Enter image snapshots" title="Direct link to Enter image snapshots" translate="no">​</a></h2>
<p>That's when we decided to experiment with image snapshots. Since we already run our frontend E2E tests with Cypress, we explored the various <a href="https://docs.cypress.io/plugins#visual-testing" target="_blank" rel="noopener noreferrer" class="">Cypress plugins</a> available for visual testing. We settled on using <code>cypress-image-snapshot</code> because of its simplicity, flexibility in adjusting the matching threshold (more on that later), and its ability to generate a heatmap of differences between actual and expected results.</p>
<p>Interestingly, this plugin uses the diffing logic from <code>jest-image-snapshot</code>, so if we ever want to add image snapshots to our unit tests, we could do so easily (and without bloating our <code>/node_modules</code>). The plugin is well documented, and there are <a href="https://www.useanvil.com/blog/engineering/cypress-snapshot-testing/" target="_blank" rel="noopener noreferrer" class="">resources available</a> on how to get started with it. In this blog post, we'll explore how to maximize regression detection without making our tests too flaky.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="setup">Setup<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#setup" class="hash-link" aria-label="Direct link to Setup" title="Direct link to Setup" translate="no">​</a></h2>
<p>Setting up the plugin is pretty straightforward, as described in its <a href="https://www.npmjs.com/package/cypress-image-snapshot#installation" target="_blank" rel="noopener noreferrer" class="">documentation</a>. Here's the quick rundown:</p>
<ul>
<li class="">In <code>cypress/plugins/index.js</code>:</li>
</ul>
<div class="language-js codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-js codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  addMatchImageSnapshotPlugin</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">require</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"cypress-image-snapshot/plugin"</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token function" style="color:#d73a49">addMatchImageSnapshotPlugin</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">on</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> config</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<ul>
<li class="">In <code>cypress/support/commands.ts</code>:</li>
</ul>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">import</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"> addMatchImageSnapshotCommand </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">from</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"cypress-image-snapshot/command"</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token function" style="color:#d73a49">addMatchImageSnapshotCommand</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<p>Then, in your spec file, you can use the plugin's command to take a snapshot and assert that it is not "too different" from the reference snapshot. If there’s a mismatch, the plugin generates a diff image with the reference on the left, the actual result on the right, and a heatmap in the middle highlighting the differences. The snapshot command takes two parameters: a name for the snapshot and some options. We'll dig more into the options below and how they relate to "how much different" the actual and expected snapshots are. For example:</p>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">cy</span><span class="token punctuation" style="color:#393A34">.</span><span class="token function" style="color:#d73a49">matchImageSnapshot</span><span class="token punctuation" style="color:#393A34">(</span><span class="token string" style="color:#e3116c">"EggsBoard"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  capture</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"viewport"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  comparisonMethod</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"pixelmatch"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  customDiffConfig</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    threshold</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  failureThreshold</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  failureThresholdType</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"percent"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><br></div></code></pre></div></div>
<p>The <code>capture</code> option comes from Cypress' <a href="https://docs.cypress.io/api/commands/screenshot#Arguments" target="_blank" rel="noopener noreferrer" class=""><code>.screenshot()</code></a> command and it has 3 modes:</p>
<ul>
<li class=""><code>runner</code> will capture the whole runner window, which includes the command log. This isn't particularly useful for E2E snapshots as we’re not interested in testing the command log, and there could be a lot of non-stable info in the logs such as randomly generated guids.</li>
<li class=""><code>fullPage</code> is the default and will try to capture the whole page. It seems that if the page is taller than the viewport, Cypress will scroll the page, take as many screenshots as necessary and stitch them together. In my experience, this only works well if the scroll container is the <code>&lt;body&gt;</code> element. If you have for example a non-scrolling sidenav, and a main section that scrolls, Cypress won't magically scroll the main content and stitch it.</li>
<li class=""><code>viewPort</code> will capture the viewport of the tested application. In practice, there seems to be little difference between this and <code>fullPage</code> if your <code>&lt;body&gt;</code> doesn’t scroll, but you may need to adjust the <a href="https://docs.cypress.io/api/commands/viewport" target="_blank" rel="noopener noreferrer" class=""><code>viewportWidth</code>/<code>viewportHeight</code></a> settings in Cypress to avoid unwanted scrollbars in your snapshots.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="fiddling-with-the-diff-options">Fiddling with the diff options<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#fiddling-with-the-diff-options" class="hash-link" aria-label="Direct link to Fiddling with the diff options" title="Direct link to Fiddling with the diff options" translate="no">​</a></h2>
<p>Now, let’s talk about comparison methods and thresholds. These options are provided by the underlying <a href="https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#%EF%B8%8F-api" target="_blank" rel="noopener noreferrer" class=""><code>jest-image-snapshot</code></a> package.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="comparison-method">Comparison method<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#comparison-method" class="hash-link" aria-label="Direct link to Comparison method" title="Direct link to Comparison method" translate="no">​</a></h3>
<ul>
<li class="">The <code>pixelmatch</code> comparison method works in a straightforward way: you give it a threshold between 0 and 1, which will be used to tell, for every pixel, if the actual pixel matches the expected pixel. A threshold of 0 requires an exact match in color, while higher values allow more variation. Then, you specify a threshold for the whole PNG: how many pixels you accept to be different (either in absolute numbers or as a percentage of the picture). Here's an example where we altered some figures in the dashboard data, causing the graph widgets to differ:</li>
</ul>
<figure><p><img decoding="async" loading="lazy" alt="Pixelmatch diff" src="https://engineering.pigment.com/assets/images/data-changed.diff.pixelmatch-4afe2c11c6b3d67c943f83717ccc02f5.png" width="2700" height="1150" class="img_ev3q">
</p><figcaption><em>A diff generated by the <code>pixelmatch</code> comparison method</em></figcaption><p></p></figure>
<ul>
<li class="">The <code>ssim</code> comparison method, as described in <a href="https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#recommendations-when-using-ssim-comparison" target="_blank" rel="noopener noreferrer" class=""><code>jest-image-snapshot</code> the documentation</a>, « calculates differences in structural similarity by building a moving 'window' over an images pixels ». It is supposed to reduce false positive and be more sensitive to changes in the image. You can read more about the technique in <a href="https://ece.uwaterloo.ca/~z70wang/publications/MAD.pdf" target="_blank" rel="noopener noreferrer" class="">this paper</a> or on <a href="https://en.wikipedia.org/wiki/Structural_similarity" target="_blank" rel="noopener noreferrer" class="">Wikipedia</a>. Here is the diff with this comparison method, on the same modified dashboard:</li>
</ul>
<figure><p><img decoding="async" loading="lazy" alt="SSIM diff" src="https://engineering.pigment.com/assets/images/data-changed.diff.ssim-fb3b79a0abcb7a7307c87dc5827a3f08.png" width="2700" height="1150" class="img_ev3q">
</p><figcaption><em>A diff generated by the <code>ssim</code> comparison method</em></figcaption><p></p></figure>
<p>The red areas are more fuzzy, indeed more tailored to the area that changed structurally. The pie chart in the middle seems to not be detected as changed, whereas the pixel-by-pixel comparison above did show some differences. Overall you could argue that this method is less helpful to show what has changed, meaning it could take longer as a developer to find out the cause of the test failure; depending on your use case — whether you're testing graphs, text, or cat pictures — this trade-off may or may not be worth it.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="thresholds">Thresholds<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#thresholds" class="hash-link" aria-label="Direct link to Thresholds" title="Direct link to Thresholds" translate="no">​</a></h3>
<p>To illustrate the threshold options, let’s use an example where a 1px margin is introduced at the top of the dashboard (a plausible regression when refactoring page layouts). The per-pixel threshold is configured through the <code>pixelmatch</code> option:</p>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">  comparisonMethod</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"pixelmatch"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  customDiffConfig</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    threshold</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0.1</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// &lt;- that's the per-pixel threshold (defaults to 0.01)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">,</span><br></div></code></pre></div></div>
<p>The first diff below is generated with a threshold of <code>0</code>, meaning the slightest difference in pixel color will trigger the pixel to be off. The second diff is generated with a value of <code>0.1</code>. You can clearly see in the gradient area at the top of the dashboard that the higher threshold allows for some leeway, and in the second diff, you can see part of the extra margin-top shown in red at the top of the diff:</p>
<figure><p><img decoding="async" loading="lazy" alt="0.0 threshold diff" src="https://engineering.pigment.com/assets/images/top-margin.diff.pixelmatch-0.0-eddcbf297129331180da6647d8c6b7c7.png" width="2700" height="1150" class="img_ev3q">
</p><figcaption><em>A <code>0.0</code> threshold (aka pixel-perfect) detects 88917 different pixels</em></figcaption><p></p></figure>
<figure><p><img decoding="async" loading="lazy" alt="0.1 threshold diff" src="https://engineering.pigment.com/assets/images/top-margin.diff.pixelmatch-0.1-678c17973ceb6168f897505c4cfc2fe8.png" width="2700" height="1150" class="img_ev3q">
</p><figcaption><em>A <code>0.1</code> threshold detects 9531 different pixels</em></figcaption><p></p></figure>
<p>The global threshold controls how many pixels must differ before the test fails:</p>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">  comparisonMethod</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"pixelmatch"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  failureThreshold</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">0.03</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// &lt;- that's the global threshold: how many different pixels makes the test fails</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  failureThresholdType</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"percent"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// either "percent" or "pixel"</span><br></div></code></pre></div></div>
<p>As a final word in this section I would like to mention that it is possible to catch a one-pixel regression: for the diff below, I've set a random pixel to <code>#E0E0E0</code> instead of white (feel free to open the image below in a new tab if you don't see the red pixel in the zoom box):</p>
<figure><p><img decoding="async" loading="lazy" alt="one pixel diff" src="https://engineering.pigment.com/assets/images/one-pixel-diff-with-zoom-4d5da706bbaea6bc752efba6d968fba4.png" width="2700" height="1150" class="img_ev3q">
</p><figcaption><em>Error: Image was 0.0000966183574879227% different from saved snapshot with 1 different pixels [sic]</em></figcaption><p></p></figure>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="test-on-the-ci-a-new-challenge-arises">Test on the CI: a new challenge arises<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#test-on-the-ci-a-new-challenge-arises" class="hash-link" aria-label="Direct link to Test on the CI: a new challenge arises" title="Direct link to Test on the CI: a new challenge arises" translate="no">​</a></h2>
<p>After fine-tuning our options, we were confident enough to try this on our CI pipeline. However, to our surprise, there was a consistent difference on the CI compared to my local stack. A fairly small difference, and you'd be hard pressed to notice it without the diff heatmap, but big enough that it would prevent catching a lot of regressions if we decided to lower the threshold:</p>
<figure><p><img decoding="async" loading="lazy" alt="Local vs CI" src="https://engineering.pigment.com/assets/images/local-vs-ci-ssim.diff-7437924406682da015921d9d9fd7d7bb.png" width="2700" height="1161" class="img_ev3q">
</p><figcaption><em>A Diff between the expected snapshot (generated on my local stack), vs the actual snapshot (generated on the CI)</em></figcaption><p></p></figure>
<p>There seems to be slight differences in anti-aliasing in the graphs, especially in the pie chart, which I haven't investigated yet. And there are also more subtle differences in font rendering, which can barely be seen even with a strong zoom in. In the image below, you can see that the top text is slightly sharper:</p>
<figure><p><img decoding="async" loading="lazy" alt="Local vs CI zoom" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAADECAYAAAAIw/HRAAABhWlDQ1BJQ0MgcHJvZmlsZQAAKJF9kT1Iw0AYht+mSqtUHOyg4pChOlkQFXGUKhbBQmkrtOpgcukfNGlIUlwcBdeCgz+LVQcXZ10dXAVB8AfE2cFJ0UVK/C4ptIjx4O4e3vvel7vvAKFRYarZNQGommWk4jExm1sVA6/owRCCtAYkZuqJ9GIGnuPrHj6+30V5lnfdn6NPyZsM8InEc0w3LOIN4plNS+e8TxxmJUkhPiceN+iCxI9cl11+41x0WOCZYSOTmicOE4vFDpY7mJUMlXiaOKKoGuULWZcVzluc1UqNte7JXxjKaytprtMcQRxLSCAJETJqKKMCC1HaNVJMpOg85uEfdvxJcsnkKoORYwFVqJAcP/gf/O6tWZiadJNCMaD7xbY/RoHALtCs2/b3sW03TwD/M3Cltf3VBjD7SXq9rUWOgP5t4OK6rcl7wOUOMPikS4bkSH6aQqEAvJ/RN+WAgVugd83tW+scpw9Ahnq1fAMcHAJjRcpe93h3sLNv/9a0+vcDJa5yiKmy6YEAAAAGYktHRAAAAP8AAEfbj5IAAAAJcEhZcwAALiMAAC4jAXilP3YAAAAHdElNRQfoCRkPAQea4ar0AAAAGXRFWHRDb21tZW50AENyZWF0ZWQgd2l0aCBHSU1QV4EOFwAAEvFJREFUeNrt3Wd0VdXWxnEjEVJIAtJDh0gRKYlIUbo0UeHKpUgTAWmiIFwQKaFIE+QC6kUpIiiigDQJNTRpUgQREUGkE6qRGpJQQu7nZ77v2CfnJiEnJ//ft2cc2GftwpnsMcdayyfpQXLyI3BkL5CPi8/dPkC6D8Czr5+Xn37GX39zgXx8nD/nAvN8puoEfdL2AqXn4X3TYbxp7lGfTP4A+aTz3/d5BADw/3jgULAe5fIAALIiCiAAgAIIAEBW4evjk/oW0//pEbpqurv59wEA+F/Y+uPDGyAAIKujAAIAKIAAAGQVPsnJdNwAALwBAgBAAQQAgAIIAAAFEAAACiAAABRAAAAogAAAUAABAKAAAgBAAQQAgAIIAAAFEAAACiAAABRAAAAogAAAUAABAKAAAgBAAQQAgAIIAMjSfLkEjzxSsdoLkn8/+me6fl+DujUlb4iaz00AAN4AAQCgAAIAQAEEACCt0ANMgXJlSksODs6ZquM9We4JLioAj7RwySrJHbq+I7l5s4aSly+cwRsgAAAUQAAAKIAAAHgeeoAp8NGHIyQ3rP8cFwUAeAMEAIACCAAABRAAAE9FDzATSEhIlPzu8A8kL1q6WnLinTuSu3RsJbnWs1Ulv9q5r2R35/l4+vjs2q5DR02WvG3nXsnx8Xo+ZcJKSB7Yr7vk19q3fKjPw40btySPGDtV8rKV6yVfvXZdcumSxST37NZecp8endwaT3JysuRp0+dK/nzeIsmnzpyTnOfx3JJbNm8ieUzkAMmu5uH+Z+ZXkvsNel9y106tJc+ePsGt8w1/9iXJv/52VPLOjd9JrlEt3K3xRQ5+S3L+/HklT5wyU3Ls31cl13mumuSZH4+TXKxoqOTe/YZLnjV3oeP5r1yzUXK24DDJcz6dKPn1jv/kDRAAAAogAAAUQAAAHj56gJlAx279Ja9YtUFyQIC/5GpPV5a8eJn24H7Yvturx3f9+k3JdZu2k3z1qvbEXmxaX3KR0IKSl6xYJ7lLr3cl+/v7SW79SrM0vb537tyVXL+Z9uwOHjoiuWiRQnr+tapL3rl7v+S+A0dLvnw5VvL7kf0dx9env86TnfnFt5Jtz87uh/nb78ck2x7Zjl37JO/avERy9uzZverfe9TazZLPX7gkufozVSTv3vuL5OhN2yW37fy2uX5LJdeoFiF53oJlku/eves4XtsDDytdnDdAAAAogAAAUAABAMh49ABTYMB7YyWHhASl6nh2XlO92jUkHzh4WLLtqeXIoT2Pn7atkGz3L7x5M07yM3VbpGr8nj6+n37+VbLt+T3zdCXJKxfPdjxe5w46j2ndhq2SixUJTdfnb85XiyXbnl945QqSd2zQP+/nl0PyyVNnJVes/oLk71askTx8cB/Jx0+ekWx7foGBAZJ/3hEluWSJopITE3VeaK1GbRyft7lfaw+rZ9d2XvV7c+bcecm/74uWnD9fHsmnTuu8yvJPN5a8d99BPf7Z8+b5bun479fVfoBzZ0ziDRAAAAogAAAUQAAAPA89wBQ4fOTPND1e7N/XHD//Yfsex8+fr/esZNtTs+w8rA5t/yF59PiP3Bq/p4+vTFhJfch99THff+A3yW//a5Tkhg10v0c7b3Hk0H4P9fmz87os26O0PT+rlFkL9NIJvZ9BQc5rba7f6Dyexg1qSbY9P8uO184rsz3A9Ru2Sfa2HmDDevr82Z6fZa+vXev16LETki9f0XmexYsV5g0QAAAKIAAAFEAAALwPPcAUWP/9l5Ib1n8uXb8vNvaq4+elShRL1fFtj8Dbxmd7Gt/O0x7ikJE6b+nT2V87ZsuurTnnU93/0FXPy11/ubjexYulbh6iq56fu/c/tT0lV+dzJfZvr/69CcoZmKq/nz37Y46fP3jwgB913gABABRAAAAogAAAeD96gB4oKNi5J3Ph4uVUHT/G7C/mbeOzWjZv4phPmLUtd/+k+6tNnzVf8tYdOm/uTbMf3trlc9N0/PnyPu74+ZmzF1J1/HMxFyUXDi2g/0t+VP+fnNfFeE6fiUnVeFydT748zt9vx2vdcbG/nStXr93gR4o3QAAAKIAAAFAAAQDwdPQAPZBde9LavG2X5IuXrkguVDC/5Hv37klevnK9V49vw+Ydkj+ZofM4n6tZVfLg/j0lly5VXLLtwb3wShfJZ89dSNfnofHztSVHrdkk+csFuj9e99fbSna1H2ClGs0kBwbofn5njmx3HI8Vba6/7bHa65uQkOh4PlbTxnUdPy+QP6/j53Yt2+vXb0rOlSvY8XmKOX/Ru4uCbzbHz2/FxfEGCAAABRAAAAogAACejx5gCvQb9L7k4OCcqTpe5afKS57x8VjJ9WrrWpM1q0dI3rXnZ8nhz74kuX6dmpIPHjoiOelBUqrG7+njCy2k89iiN2kPZ/W6LZK37dgrOc/juSSvif7B8fs6d/xnuj5/3V5rI/nzeYsk2/3yyoY3lFyuTCm9P3sPSLY9uP59ukq2PcQK5Z+QbPfjm/nFt5IjajXX56VauGS736adR1ql0pOSu7i43nY/vdy5QySfN/NMwyrXl1y0cCHJp8/qvMaKFcpKPnT4D6/6vbP319pq/r08/2JHyQP7dZf8goueLW+AAABQAAEAoAACAJAu6AGmwNFjJ9L2oruYZ2PXMoxaPFvyv4aOk/z96o2SV63bLLldq5cl16lVTXLnHgMlZ8v2aKYen+1hbFmzQPLIcdMk79i1T7Kdl1iuTGnJ40cNktyjy6vp+vzlyJHdnM83kiPHTJG8PCra8fzCzDy8Xm90kNzbZFemT9Ue+RNhJSXPnrtQsu0h5c2TW3KfHp0kj4kcIDl79uyO4wkJCZK8cpE+n/3e1fEePnJMclKS9qCXfP2p5CmfzJHsbT3A8mXDJE+bpGvdfjhtluSdu/dLbtxQ54nSAwQAgAIIAAAFEACADOGTnJyczGXwbJevxEq2+7dVjajo1vFGjp0qeeyk6ZLf6aNrXf57wrBMPT4A4A0QAAAKIACAAggAQBZDD9ADxcXdllwuopFku79evdo1JNt5XmdjdL86u7+Zv7+f5L1bl0u284I8fXwAwBsgAAAUQAAAKIAAgCyOHmAmcPqM7kdm17LcuGWn5L9ir0rOFRIsuVbNqpJHDHlbst1/LbOPDwB4AwQAgAIIAKAAAgCQxdADBADwBggAAAUQAAAKIAAAFEAAACiAAABQAAEAoAACAOC5fDPiSx+YmYeP+nAjAAC8AQIAQAEEAIACCABAGsmQHqAPPT8AAG+AAABQAAEAoAACAJBeMqYHyHUHAPAGCAAABRAAAAogAADpxTcrnnRi4h3J8+Z/J/nM2Zh0/f5KFctLbtemBU8iAPAGCAAABRAAAAogAABpJUv2AO8nJUn+YftuyctWrE3X73+rV2fJ9AABeIptO/ZIHhI5UXJE+FOSJ44bIjnA3583QAAAKIAAAFAAAQDIeL5cAte+mfex5Nq1qqfqeH5+ObioADxS4h2dJ/3T/oOSixYNlfzgQTJvgAAAUAABAKAAAgDgeegBpkBwSJDkAvnzPtTvTzLzFtes2yJ5xuyvJcfdvi3ZzjOsXi1c8tAROs8nrHQJya7m+Xj6+G7F6ffNmvON5LXrdbwJCYmO39ere0fJNcx4fXzSd8dLu5btoiVRkhcvWSX5+o2b6Xo+ycnaA9q994Dj/T9+4rTkXCHBktu0ekly21YvS3bVQ5//zTLH72/auK7kyCH9HI8Xn5AgefCwCY7nM2XiCMnly4W5Nb4e3dpLLl2quOQPp8xwvL9NG9eT/Fbv1yUH5Qx0HM/kaTMdr8ePu/ZJbtRMx/tmz9ckd2rfkjdAAAAogAAAUAABAHj46AF6INtTsT2rYSMnOf79kiWLSZ40VXsG9evUlHzy1FnJIaYnY+f5ePr44m7HS36z7zDJtmcWUUXXNixbprTk7T/ulbxl64+SP/tkvOQXmzZI0+fBns/b/bXHtGDhcslFihSSnC/P45LXrNssOWr1BsnjRr8r2fYIbU9wzrxFkodEfuB4PqXM/T9y7Ljk7m8Olhy9aZte74/1egcH5ZR8Ky5Osp3HVv2ZKm5df/t8/RV7VfKp0+ck37171/F4rsZXokQRyfsm/ao/2r76s33x4mXJu/b8LPnEyTOS/zN1DD+yvAECACiAAABQAAEA8H70AFNg0JBxkkcHTkvV8SaM0R5HHbO26B/HTkj+zMwTym/mIX41Z6rkalW1x3Ht+g3Jfd6JlGx7GOGmJ2Z5+vhOmHlZtsfS4uXGkr+YOVmynSf1y8HfJc+b/51+oemJ2nmR2bJlS9XzsmnzDsnfR613PJ9Z07UHlztXiOQDBw9L7vzGAL1fC5ZKfvEF557mnHkLJefJk9ut+3/zlvbEevcdKvm7paslN2pQW/LrnVp71e9NTMxFyetWzpdcskRRyXv3/SL5tW79JbvqCdp5egUK6L/fl17pIvnZmlUlf/6Z9vxzBgbwBggAAAUQAAAKIAAAGY8eYAocOXo8TY9n99uyTpqe19E/9Pu7vf6q5IgqFR2PZ3tAHdu9InnJstVujd/TxxcQoGuB+vv7ST7021HJS5evkdz4+TqSKz5VVvK0ySPT9Xmz8yy3bN0l2c4LbNe6ueP1tKpUelLy1g2LJecM1B7oY4/pz8Qas3bq/gOHUnX/7Ty+1zq0krzWrC27bcceya+a88/sng7X62V7ftaT5cvo/a1cQfIvpud7//59ftR5AwQAUAABAKAAAgDg/egBpsCq5XMl2x5RWjt95pzj5/5mPzRfX/fmmYUWLCDZVY8hs43P7nf3zlvdJI8er/M4e/R5z/F4BQvkk9z8pUaS7X5yqd0v8na87j936cpfjn8+Z1CgW8e3a3m66hk+7Ptf0Fw/O6803uzXeN/MuwR4AwQAgAIIAAAFEAAAeoCeqGD+fI6fJyTqPML797UH4qrncuGS7h/m7lqbnj4+2+Pq3FHnlbVt9bLkI2YeY/RG3X8uavVGyXb/w0Rzvna/NT/TE3PFL4f+eVc9Ont9M/r5dLV/pL0/1o2btyTfNBngDRAAAAogAAAUQAAAXKIH6IFKlyouuVTJYpJ37ton+dBhXdsy3KwFaHtUtqflbeP78/gpyT/u3i+5YMH8kps0rOM4viYN60pu2+lNyUf/0P0RbQ/L3R6g7ZHatUitzVt2Srb75dm1PK9euy75jd6DHf/8R5NHuXX/Dxz8TXLMed3frmiRUMm2R7jnJ93fLvbva5LLlSkl2e4/Z+eBWjEXLkm2+xHatUljY/+WfOrUWX6keAMEAIACCAAABRAAAE9HDzAFBg0ZJ3l04LRUHS8iXOexTRw3RLLd3+uV5k0k//uj2ZJbtNK1LguHFpRse1Jly5SWHBpawK3xe/r47PHGTPhYH3rTY+v5RgfJpUpoT+v7VdGS7bzEJo20R5jn8dxp+vw1b6Zrjy5dvlbyF18uknz5Sqzk5+s/J3nJMt3/MHqTznvs3rWd4/nY7Or+d+kxUHL7ti0k2/0Zv/52ueSnKmgPtF2bFo7Xy96/ymb/wxUr10u+Yq5XubJhkg/++rt5frz7Z7NwIf33+URYScl2f8HIUR9KbmPm2dasHsEbIAAAFEAAACiAAAA8fPQAU+DI0eNperyiRXUelF070c7DGjmsv+QyT+g8qAULV0i+d++e5E7tW0quW7uG5K49B7o1fk8f39PhFSV/9on2cN8b/oHkwcMmOB4vJDhIcq/uHSWPjhyg/6jc3P/OlSJFCkmeNV3HP2joeMmr126SvGhJlOP5DH23jx5vQC/H+20NH9JXsp33OGee9ih7vT3U8Xh1a1eX/OH44ZJtj9iy8wA/GKP7Pb4zaLRkO0/04qUrkseNGiTZznNdG/2DV/3e2evbu0cnfd7HTpU8feZX+nyF6PNFDxAAAAogAAAUQAAAMoRPsl2IDx7Prp3p7lqTm8zaka3ba8+nbevmkj+Z8r5kVz0uTx+fZecN3rlzV7Ldj89VTyyjxd2Olxwfn5Ch53Pv3n3J167fkBwQ4C/Zru2Z3uIT9Pr4+/npj6SL/QuzmqSkJHP9EiUH5QzkDRAAAAogAAAUQAAAMh7zAD2QXcvxrf6Rko/9eVLy5A90nlTViEqSY2OvSp45Z4Fk2zOq/kwVfUhMT83Tx+cuOy8us7M9tIfdU7NsjzF/vjwedb0C/P350XFDtmz67y0z9fx4AwQAgAIIAKAAAgCQhTAP0APZW2LXduw7cJTkmJiLbh3f9oS6dm4refQIXdsyMCAgU40PAHgDBACAAggAAAUQAJDF0QPMhG7H67y4ffsPSY6Li9P/5Zh5O5UqlJNcuHDBLDU+AOANEABAAQQAgAIIAICXowcIAOANEAAACiAAABRAAAAogAAAUAABAKAAAgBAAQQAgAIIAAAFEAAACiAAABRAAAAogAAAUAABAKAAAgBAAQQAgAIIAAAFEAAACiAAABRAAAAFEAAACiAAABRAAAC81n8BWLVHJ9kZ85oAAAAASUVORK5CYII=" width="448" height="196" class="img_ev3q">
</p><figcaption><em>Differences in font rendering between Ubuntu and Windows</em></figcaption><p></p></figure>
<p>The root of the issue is that Chrome delegates font rendering to the operating system, and in our CI environment runs Ubuntu Docker images, while our development laptop runs Windows.</p>
<p>To work around this issue, we added an optional CI job to regenerate image snapshots when needed. When a legitimate change to a snapshot occurs, we trigger the job manually and we save the newly generated snapshot to our git repository (<a href="https://git-lfs.com/" target="_blank" rel="noopener noreferrer" class="">git lfs</a> is our friend!). While this adds about a minute of manual work, we found it worthwhile as it allows to keep a very low threshold for our image diffs, and we don't need to update the snapshots very often anyway.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="concluding-remarks">Concluding remarks<a href="https://engineering.pigment.com/2024/10/07/snapshot-tests-with-cypress/#concluding-remarks" class="hash-link" aria-label="Direct link to Concluding remarks" title="Direct link to Concluding remarks" translate="no">​</a></h2>
<p>Looking back on the process of integrating image snapshot testing into our E2E tests, I found it more trial-and-error than expected. It took several hours of tweaking options and thresholds to find the right settings for our needs. That said, the flexibility allows you to tune the system to match your requirements — whether you need pixel-perfect accuracy or some room for visual variance.</p>
<p>In addition, the approach of image snapshots does come with a couple of drawbacks worthy of mention. It adds a bit of maintenance, as every now and again a legitimate change will require you to regenerate and update some snapshots. Furthermore, after a few months of using this in our CI pipeline, we've had a few bumps of Cypress' version that changed slightly the size of the snapshots, for some reasons. Another aspect to take into account is that it is more difficult to track the cause of a regression from an image diff than from an html diff that you could get with jest's snapshot testing.</p>
<p>But overall we feel that the use of image snapshot testing in our E2E tests suite is a great addition, giving us more confidence that we are not unintentionally introducing visual regressions in our applications.</p>]]></content:encoded>
            <category>Front</category>
            <category>Testing</category>
        </item>
        <item>
            <title><![CDATA[From Zero to Millions — Inside Pigment’s Tech Team, Real-Time and Scaling Challenges]]></title>
            <link>https://engineering.pigment.com/2024/08/28/from-zero-to-millions-podcast/</link>
            <guid>https://engineering.pigment.com/2024/08/28/from-zero-to-millions-podcast/</guid>
            <pubDate>Wed, 28 Aug 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Our R&D team is hard at work tackling the challenges that come with building a modern business planning platform. Engineering Program Manager Chaimaa E. and Tech Lead Benjamin Hewins, recently spoke with Elliot Kipling and Enginears to share their insights on Pigment's engineering team, culture, and the technical problems they're solving each day.]]></description>
            <content:encoded><![CDATA[<p>Our R&amp;D team is hard at work tackling the challenges that come with building a modern business planning platform. Engineering Program Manager Chaimaa E. and Tech Lead Benjamin Hewins, recently spoke with Elliot Kipling and <a href="https://www.youtube.com/@Enginearsio" target="_blank" rel="noopener noreferrer" class="">Enginears</a> to share their insights on Pigment's engineering team, culture, and the technical problems they're solving each day.</p>
<p>Listen in to learn more about Pigment engineering 👇</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/jUjhTrDDIbQ?si=6HV07Xu1NiDKdoju" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin"></iframe>]]></content:encoded>
            <category>Podcast</category>
        </item>
        <item>
            <title><![CDATA[Fuzz-driven slow path detection]]></title>
            <link>https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/</link>
            <guid>https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/</guid>
            <pubDate>Thu, 27 Jun 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Fuzz testing consists into executing a given piece of code against randomized inputs. It is a known tool when you want to detect bugs in your algorithms, but we rarely talk of it for performance related topics. Let's see how we can turn fuzzers into tools able to help us into detecting slow code paths.]]></description>
            <content:encoded><![CDATA[<p>Fuzz testing consists into executing a given piece of code against randomized inputs. It is a known tool when you want to detect bugs in your algorithms, but we rarely talk of it for performance related topics. Let's see how we can turn fuzzers into tools able to help us into detecting slow code paths.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="performance-is-not-an-option">Performance is not an option<a href="https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/#performance-is-not-an-option" class="hash-link" aria-label="Direct link to Performance is not an option" title="Direct link to Performance is not an option" translate="no">​</a></h2>
<p>At Pigment, performance is not an option: we build pivoted grids able to render millions of cells on the screen with real-time updates. It consists into ingesting an array of all the rows to spread them into the relevant buckets.</p>
<p>You may imagine something starting from:</p>
<p><img decoding="async" loading="lazy" alt="Flat grid" src="https://engineering.pigment.com/assets/images/flat-grid-ff7b1815b590cc5d369946fe811ea8f3.png" width="413" height="202" class="img_ev3q"></p>
<p>And resulting into something like:</p>
<p><img decoding="async" loading="lazy" alt="Pivoted grid" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARkAAACWCAYAAADqgfyuAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABm7SURBVHhe7Z0HWFRX+sbfiICAiooFFZUgKhas2Gvsiia6xq5J1uQffKIpm0dNTNbdbEwxuslmU1xNUWMXUSzEGhuKvTesWEAEEcQCoqD53++bO8jgIEO5Mxfm++WZZ+aec2fiYea895xzv/O9z/2pAEEQBI0ooT4LgiBogoiMIAiaIiIjCIKmiMgIgqApIjKCIGiKiIwgCJoiIiMIgqaIyAiCoCkSjCfkm/iEJPWVYK9UqVRBfZUzIjJCviGRcS/jph7ZH7fvpth9+y0RGZkuCYKgKSIygiBoioiMIAiaIiIjCIKmiMgIgqApIjKCIGiKiIwgCJoicTJCvimMOJmH6enYvfco9hw4jpIlHdCnRwc09PPFc88Z6h88eIgNf0TgZOQFuJctjcEDeqJKZQ9DpcL1uASs3bAD8TcSUad2LfTv0xluri5qrQGqW75qE5o0qoeO7ZqrpQWnMOJktGw/9ewLUVexfvNO3L2XiqaN/dDjhTZwcnTk+oIicTJCkWD7zoO4EnMdbwcNx6ghgdi0ZTcuXYnhOuokYUoHupeSivfeGoUuHVticcg6JCYlc33SrdtYFroR7Vo3xaT3xqCce2mErNqMR48ecz09b9mxD/OXrlU6WQrupz3gcj2hZfujlc9du347Bvbvhr+NG43k5DvYvHUP11kTERnBZjx8mK50lGR069Qapd1cUdWzEho1qKNcfaO5PjHpFq5dv4HAXp346kxX+OpVK+P02SiuT7h5CzWqe3K5s5Mj2rZqymJCnZLIePSIP/edoBHwqubJZXpC6/ZHx8ajeZP6fI6rSyl0bNscscrIJ00ZHVkTERnBZjgpHWPYoD7wql6Fjx8ponA9PgHVq1Xm45uJyXBVOpdx+E9TCJoSXL4Sy8f16nhj0IvdM6cWCTeTuDO5uDjzMXW81gH+cHZ24mO9oXX72ysjHBr9GLkef1MZ7ZRR/i7W/XuIyAi6gKYGW8P389W9Tu2aXEbrFdQhHB1L8jHh4GD+J5uYdBuha7cqV/MmhbbmYE20bv/lq7HK1OyAIjzNMkXJWojICLrg6IkziFSmAS+/1COzk5QoUQIZGRmZaww5QWstIas3oWWLRvCr66OWFi20bD8LUNgW9O3VEdWqVlJLrYeIjGBzTpw+j+27DmDkkH4oX66sWgp4VCjHV/YHD5+sIcTFJ/JdFiPUwZaGrIePtxc6tw+w+lW6MNCy/beS72BRcBi6dGgJ/wZ11FLrIiIj2BQaxv+xfS+vTXhUcFdLDdBxSWWqcCryAk8n6Ip8UumQjRr4cj2tYaz+fRsqV6qArp1aFUmB0bL9JEDBoRvRsnkjNPX3U0utj8TJCPmmoHEyD5Sr9C+/rUDUZcMtWyPUIUYP68evqWPNWRCKlNRU7mjdOrdGx3YtuENR7Mi8RauV8ic/4VKlnBH02suoWaOqWmJgwdIwji/p2bWtWlJwChono3X76fb9uk071RoDNFJ6O2iEyWgov1gaJyMiI+Qbayatoquyk3JVd3BwUEtsT2EE41mKXtsvwXhCscFFuULrqYNZm6LcfhEZQRA0RURGEARNEZERBEFTRGQEQdAUERlBEDRFREYQBE0xGycjzoCCIFhCvoPx7N0Z0FKsGYylRywNxiquUD+R9kswniAINkZERhAETRGREQRBU0RkBEHQFBEZQRA0RURGEARNEZERBEFTHD5RUF9nkpJ6H6V0aiOhJyizmaV/J3L6WxKynjOVHTtxFhXKl1Me7kUyZaQRan9pN1O3RmtCnkVfffM/nIo8j1YBTdTSpyGTs8+mfY9vv/8Vt5Jvw7+hn4kDwMlTZzFl6jf4Ze4STt7tV7c2P+cG9RNbtf/y1Rh8/tUP3KbzFy+hsX99uLiUUmtNsXX7ZSRjBY4cP4N5i1dzrtV3xo5E7+4dELp2CyL2HVHPEPLKjp37MCZoImKuxbEzYk5Qp/rHp1+jV49OWLbwR07+9N8f5yAj4xHXU2f99Mv/4q3/G4XffvkGZ89fxLIVa01SWuoN+jdP+ugLDBrQByuXzkbzpv74+r8/s6VtdvTQfhEZjaG0iXv2H2MP42aN/Ti3al3fWhg5NBBHjkVmuv1RLtewjeFYtnIjZ69//NjwJVMeV/Iz3rhltyJKR7lMANIz0jFn9gx07dJOLTHPoSMnUMf3eXTr0h5lSrvhL0rHvBoTi2vX47h+8x87MeDFXmjSuAHKuZfFKyP+gl27D+DO3Xtcr0dKu7nh0ynvo02rZpzTt33bABYT428pK3pov4iMxiQkJLGRulc1g0ugEbIbpYTOZE8aHROHWXOC2d2vQ5tm2K2IScRewygn9noCgkM3oaSDA6p5Wt8zR690f6EDTzlz49Tpc/BvVE+ZlhrmpdSRPKtUwjVlBER2I1GXr/L0wEjlyhW5E8fHJ6gl+qOiR3n41TM4FtCI4+ixUyhfzp1/S9nRQ/tFZDQm+c5dtkl1djZYh5rDq7onJrzzGgsMWZS2auHPVhlGyM+4W5fWeL5WdbVEsJR791LhUaG8ekRWr8/BoYQhVy5ZilAnzdo5n1P+y8mlUW8sDl4NJ3dfzPxpPoLeGGnWjlcP7ReR0RjyJk67/8DEoCs7jx8/wvGTZ/Hjz0vx6fTZWLHmD7XGgCWLcIJ5aBSZev++emQKdTialppbyygKjBjyEh7evoDJE8dh8pRpuBYbr9Y8QQ/tl1+vxtDQNvV+GhKTktUSA7QGM3fhKp5H7z14AqciL2LUkEBMmRiEIQN7qmcJBaVB/Tq4dClaPQLuK98F3ZWi9Qlaz6iuTGPjE26qtYY7JuS6WFqp1yvUBnoQJBQtmvmjnjLluXLV1L+J0EP7RWQ0htZZGjesi5XK6IS+PCIt7QHCNu5A2TJucHN1RUrKfWWeXBHuyrk0fD17/jKfJ+QPWmswLoL6N/LDgcPH+S4UceDQMdxPS4OPj8HUvkO7llgdthl376Xw337z1p2oVrUyqlSuyPV6ZM++w/hAGbmQ4RsRe/0GoqKu8noLobf2Sz6ZAmBpPhma+24NP4Bt4fuU4WtJpKdnIKBZA/Tr3Znn0ewSuDCURzwOytTIq3oVnjeTi+CmrXv4MwrT+bCw0EM+GVqXoCv1xx+MV0sMZQkJiXh33Bg+3hmxHxMmf44yyndFHXHa1A/hW9ub6+jnH7zid0z7eibX0Z2Yzz+ZiEoVc2+XrfLJ0O3nX+ctxaxfF7EYpCqjj79/+DZ6dO3IIxu9tV9EpgDkNWkV/aXpKlLK2RklSphG4VEdXZmcnZxMAqX0jF6TVs34z2y+rduuTQu1hNa9HqvBY67cEbNDHffBwwc8srQUW4mMkZzapLf2y3TJitB3SwvB2QWGoDr6ARQVgdErNBWl6cLz3jXUEgO0eE7rEOY6GEELpHnpYHrAXJv02H4ZyRSAvI5kiht6HMnkdsUuTGw9kjGHHtsvIxmhWJHbFbu4o8f2i8gIgqApVpsu3VWG1nMXrcLtO6Z7ImhPj1/d59WjooVMl8StQNqfe/utJjIkLrPnLke/Xp04AMgILYQW1cVOERkRGWl/7u236nSJ5om0IEU7kY0PEhiKQqQdxmfOXcLKtVs4Opakj4LSlq/ahFVhWxF340lU4v5DJxETG48Nf0Rw/ZXo62qN4VbwpSvXzL7vYXo6v5fyumwL388r8YIgaIsu1mQepmcgYs8R3n1MmwAp3Hnrjn0cFUvpEXxr18Ti4HW8I5mg1AfBoRvh4+0Fn1peWLgsDFdVoTl64gxW/76N31ejuieH7tMuZwqIW7lmCy5eikan9i1wR7kKU46XtCK6b0UQigp5ni5t33VQffU0CTeTeI3FHDRd+n72YmRkZMDBwbALlERg+OC+PKKYv2QNRg3th/LlDKHRhh2ihvv39Dx/6RoWDgrRX7A0DI0a+PIxQaMWCssPaNaQRaV39/YsQMS12Bso4VCCR0vbdx7AqGH94OToyJ8/f8latG3VJN9rQjRdEAR7RpM1mWeJDNGlQ4D6yhQSmZ/mhWDowF6oWaOqWmqA6hYsXYvRw/rzFIqgvRf0/4o8G8UCkZKahpFD+maKTBP/uvyaMIbetw7wf+pzjBw/dQ5Llq+Hq+uTFIWpymcOH9wn83PyiqzJyJqMtF+DNRkSkWc9CgPa27M8dBPKlnYDJXb6xwdj0ai+IUnPs6A1H3pQQJI5/Oo9j79PDMKUSYbHl5+8m2+BEQTBMnQZJ/NIEYl0ZVpFIx5K1H1HGenExt1Qa3PGzdWFF5YPHjnFUyyaEtFt8wOHT6GaZ2XeBR2vLgRTWsyQ1ZtNFoYFQSh8dCkyJCyUdJvWV6ZOn405yjNtHMwNyuj1Yt8uOK1Msf41bSY+/Wo2HEuWVEYrdVDRoxy6d2mDWXOXc2KoL7/+hXdEV/J4kjVMEITCR9d7l2gkQiMOVxcXmNtU+CxoHee5EiWesiyh1lKmMCo3LkDnF1mTkTUZab8GazLWhESApj95FRiCPGjMeSLRlg6aVhVUYARBsAxdi4wgCEUfERlBEDRFbGoLgCU2tTt3H+ZAw/CIQyaPfQdPwLtmdc7zawn0nfzw0xI8/vNP1PTyVEttiy1tailpttGm9XrcDc5lm9PNgeJoU2vkSvQ1jP/bFHjXqoGqnpXVUlPEpraY07KFwZqWHuSp1OOFtvx67Jgh8KzioZ6VO7T4PfzlvmjRpL5aYr/cSEjEVKXTvDzQYNNKhmTffPcz3yjITm42rSRWRc2m1gi1YcGilYg8ezHHfXhiU2sH0EgnczNoSUeOOKbXNIKhNImWbgylBeuoyzFISLyVuaGU9mvRees27zJrUVpciY2NR9PGDdAqoCnvcwvs3ZXjn5Jv31XPeEJuNq2biqBNrZFt4bt5202/Pl3VkqcRm1o7J68bQ6MuxXBAofF94bsPoV3rpmzONW+R/Wz2bNqkAcaPfZWju4nY2Di2rKU7kdkpjja1RJzy7wtdsxGjRwyCk2POU3axqRXg4uKMgf27sajQrfUuHQPw7thR8PWpiYZ+vvDwKIebSbfUs59A7wuk3DxVaQrWhsuMvk72BI1gyM5joHI11qtNa2FD/+4lwavRu0cXk9xM5hCbWoG/9KyLbBR8uGFLBGZ8Nw9Tp8/C6TNRao0pWd9HkcvUwdKVK5M9QRYyZP/x8sBANG/aSC01RQ82rYXN0eOnkXTrNnp176SW5Iwe2i8ioyPyuzHUHiGBmfbv/6FFc38M6N+TO4w5iptNLS3wzlsYgm079mDg0DfRd8BrbOY26eMvseZ3Uw91QmxqBRPyuzHU3qA7I//7aSF8vGtgsDKKyS4wtNZgXAhvXMxsamnE+tk/J2D18l+wYM63/PjLS70xZfI76NG1A5+Ttf16sKkVkdERJCz52Rhqbxw6chxfTP8Bb46fDJcKdeFYtjZatO+Hc+cNU8uQVevYxpWoVdMLo4cPxKDhQegeOALfzZyL999+I9PIjKZZHdq2RKceg9G1z3AeIbz15iu63XZCgupetgw8KpTLfJAjKY1MaCsNkbX93jpov643SOodrTZIFmRjqDXR6wZJe7GpzQmxqRVypSAbQ+0dPdq0WhM9tl9GMgVAUj3obyQjNrViUysImpLbFbu4o8f2i8gIgqApIjKCIGiKiIwgCJoiIiMIgqaIyAiCoCk53sIWBEHIDUtuYUucTAGQOBmxRJH2S5yMIAg2RkRGEARNEZERBEFTRGQEQdAUERlBEDRFREYQBE0RkREEQVNEZKzE9bgE/DQvhNNqfjdrEc5fvMpGboKQHygZ+PRvZnEi8YkffcFWtDlBdXQOnfvDrN84mXhWyKY26O2P8NLgN7AsJCzTXbKwEJGxAkeOn8G8xas5fy9Z1Pbu3gGha7cgYt8R9QxBsBwSiS9nzER9P1+2nu3ZvSNb0ZIlbXbEptYOoFy9e/Yfw+ABPdnAjSxq6/rWwsihgThyLDIzq3xi0m2EbQzHspUbceL0efbDIU5GXsCFqKvYuGU3W9MKQuTZC+wE2bfXC5ygqmvndnhv/JjM30xWxKbWDkhISOL8qV7VTJ3+yPmRvJUoTWJ0TBxmzQlWvuQy6NCmGdvWRuw1jHLIojY4dBNKOjigmmclLhPsmzPnLsLb2wvbw/fiX198i8XBa+Dr683OBdkRm1o7IPnOXfbKcXZ2Vkuexqu6Jya88xoLTPVqldGqhb8yjI1Va4HmTeorV6LW7JctCNHRsViweKVycYrF668Owz1l1PHxP2c8tdZCiE2tHeDqUgpp9x/gwcOcrUAfP36E4yfP4sefl+LT6bOxYo2pE6DRjlYQCHJ+bNOqGUYrUxu6QI15dSicnJx4GpUdsam1Ayp6lFe+5DQkJiWrJQZoDYZM3GhNZu/BEzgVeRGjhgRiysQgDBnYUz1LEJ6mqmcVZbrtmWnA5uTkiNKlXdkOJTtiU2sH0DpL44Z1sVIZndCXR9CPIWzjDpQt48ZeNykp95V5ckW4K+fS8PXs+ct8niCYo1XLJjh87CSuxcbz8enI8zh//lKm15LebGoln0wBsDSfDM19t4YfwLbwfcrwtSQb6wc0a4B+vTvzeg2NauYsDOURj4MyNfKqXoXnzaOH9cOmrXv4M3p2bcvPekLyydgmn4xRDD6b9j2PYkhUZnzxEVq3bMb1ZMCfkJCId8eN4eOdEfsxYfLnKKP8Vmnhd9rUD+Fb25vr6LOCV/yOaV/P5Dq6E/X5JxNRqWLu7bK0/SIyBSCvSavoL01XEfIuzu4OSXUpqansfe3oWFIt1TciMrZNWkVdl35PLqVKmfgsiU2tHUPfLS0Em7OfpTr6ARQVgRFsD4kF+aVnFQ092tSKyAhCMYKmT++M+yvHwugFERlBKEaITa0gCHaH1UTm7t0UzJ6zHNdib/Ax3WExF6FoDrr9e+bcJfVIEISihNVE5vGff+LOvRS+nUtEnotCyGrTyNacSElNw8P0dPVIEISihE2mS9HX4rBrzxFcjbmuCM1mjoalW7gUhLZ81SasCtuKuBtPohCNUNmGPyKQliUM+vjJczh8LFI9EgRBb9hEZGhhiqIKKeK1Tu2aHN68dcc+joKldAi+Stni4HW8Azkr7mXLIOpyjFJumHKR2Ow5cIyjagVB0Cd5Dsbbvuug+uppEm4mcd4Uc9y+c48zww0d2As1a1TF8VPncOzEOY5qJQw7Qg336+l5/tI1LDgUkr9gaRia+Nfl1xQBm65MnQJ7dWLBoZHNX0cN4GQ81oaC0QTBntEk4vdZIkN06RCgvjIlN5GhvRb02ZFno3hBmNZhRg7p+5TIXFNGMWvWbcerI17EtvD9cHNzzfH/qTV5jfgtbkjEr20jfm2NZhG/1KGf9cgPdKdpeegmlFWmUZTI6R8fjEWj+r5qrSmVKpbnqNiLUdEsODTdEgRBv9hkTYagoCG6Y0TTpEePHyM9I4NHOKWcnXBHGfXExhnWXbLj5OiojGjqYO2G7TxF8izE3aKCIBQ+NhMZyvJG06Jp/5mD5Nt3OMk25VehbP5zlGfaKJgTPt41FHF6DL+6PoWexUsQhMJFV7uwaVRDibdp05e5TYRGbiYmY0nIOowe1t+md5ZkTUbWZKT9GqzJaAll+qKdyM8SmKMnzmDFms3wVkZCdEtbEAR9U6TmGjRFolFOt86tEdizE3S0B0wQhBwoUiJD6y/kWeTrU/OZox1BEPSDZMYrAJasyezcfRjbdx1Qj55AaTdHDg5kCxRLoKxms+cuR+uAxmjfuqlaaltsuSbz4+z5+H39VvUIaKn8XSZPGMfR49khm9bvZs5lD6K+vV/A668OhYtLKbXWYNP6/azfEBd3AyOGDsCgAX04KDQ3ZE3GsvaLyBQAS0SGtj4YLSdoLalBvdqoX8+H8324uZbKzDifG/Qt0d6t8u5lzHYkW2ArkaG7kv/87D8YMigQz3sb4qRIFMqWKf1UHhWyaX1/0lSMHD6Ac+AGh6zlyPQJ7wXxe8imddJHX+DjSeNRq5YXi1Ftn1oYMeSlXHOyiMgUwYXf4gjF/ZA1LT0cSzrCVREWek37tihNIlnPUhqLlWu3PHOjKP3eaRtFQuIt7mT0vqvR1/m8dZt3ZWantwcovop8rKpVrcKuifSgmwDmREEPNq32joiMDXmYnoGIPUfYlpbihnLbKBp1KYZtVYzvC999CO2UqRONlOYtWm2yO704k6IIavKt29i4eQc+nDINK1atzzE3kR5sWu0dERkb4+LijIH9u7GouLm6oEvHALw7dhQvbjf084WHRzncTLqlnv0Eeh9tEiVP7R4vtOEyo69TcYeixVu3aoq6dXwwfuxrOHn6LD75/FuzToh6sGm1d+SvaWPoR5/VhpaCETdsicCM7+Zh6vRZOH0mSq0xJev7yMuJFpLTlSuzPUAjkbfefIVtP8im9b3xr7OdzJWrMeoZT6B1F1vbtNo7IjI6Ii8bRe2ZrA6JBO1nowV0c+tSerBptXdEZHREXjaK2jMXLl7G+PemIF5dFL8YdYXXafRq02rviMjoCBKWvGwUtVfq+/nyHaEXX34d3QNH4IO/T8O4oFdQvpw714esWodf5y3l1941vTB6+EAMGh7E59It6vfffiPTyKx500bo0LYlOvUYjK59hmPbjj08FbM0tEDIHYmTKQCWxMnkB0s3itoaWwbjETlZr+rNprW4InEyRRhLNooKyo/XjJGZHm1a7R0RGaFYoUebVntHREYoVuQ2YhGsj4iMIAiaIiIjCIKmiMgIgqApIjKCIGiKiIwgCJqSYzCeIAhCblgSjGdWZARBEAoLmS4JgqApIjKCIGiKiIwgCJoiIiMIgqaIyAiCoCHA/wOvYyupOI7laAAAAABJRU5ErkJggg==" width="281" height="150" class="img_ev3q"></p>
<p>The algorithm responsible to pivot the data is a synchronous piece of code executed in the main thread on user side. Having it too slow would ruin the overall user experience by freezing or even crashing the browser. So we first of all defined strict limits to avoid running the algorithm against unsupported sets of data: too many rows, too many cells and others.</p>
<p>But there was always a lucky configuration that passed and made the algorithm crazy, leading to rare complaints related to browser freezing when a grid is rendered. Each time, a new reason and a new slow code path detected and fixed. So we wanted to be proactive and anticipate these issues.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="towards-fuzzing">Towards fuzzing<a href="https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/#towards-fuzzing" class="hash-link" aria-label="Direct link to Towards fuzzing" title="Direct link to Towards fuzzing" translate="no">​</a></h2>
<p>Fuzzing consists into executing a given piece of code against <em>randomly</em> generated inputs. The approach is often used to detect bugs leading to crashes or to subtile unexpected values.</p>
<p>In our case, we wanted to push it further by leveraging it to help us into detecting slow code paths. Instead of asserting for crashes, we decided to ask a fuzzer to run our algorithm and to mark the test as failed anytime whenever it reaches an abnormal timing.</p>
<p>Our algorithm being executed in the main thread, we consider that it should never take more than 50ms to run. As written in the previous section, we do have some kill switches in the algorithm to quit whenever we encounter inputs being unsupported because of their size, so any valid inputs even too large should fulfill that requirement. As we only want to leverage our fuzzer to detect highly critical issues, our time limit is much higher than 50ms. It's rather 500ms and 10s.</p>
<p>The theoretical fuzzing approach is the following:</p>
<ol>
<li class="">Generate a valid input for the algorithm</li>
<li class="">Start a timer</li>
<li class="">Run the algorithm against the generated input</li>
<li class="">Check if the timer breached the time limit</li>
<li class="">If not, go back to 1.</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="fuzzing-in-typescript">Fuzzing in TypeScript<a href="https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/#fuzzing-in-typescript" class="hash-link" aria-label="Direct link to Fuzzing in TypeScript" title="Direct link to Fuzzing in TypeScript" translate="no">​</a></h2>
<p>Now that we have the concepts, let's see how we achieved that in TypeScript.</p>
<p>We decided to rely on <a href="https://fast-check.dev/" target="_blank" rel="noopener noreferrer" class="">fast-check</a> for the fuzzer. But fast-check alone was not enough. Indeed, as we planned to detect huge performance issues implying main thread to be blocked for several seconds or even worse infinitely, we needed a way to stop a synchronous task. So we coupled it with <a href="https://fast-check.dev/docs/ecosystem/#fast-checkworker-" target="_blank" rel="noopener noreferrer" class="">@fast-check/worker</a>. @fast-check/worker is able to spawn the code under test within workers so that it can be interrupted by the runner.</p>
<p>Our fuzzing snippet is:</p>
<div class="language-ts codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-ts codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">import</span><span class="token plain"> fc </span><span class="token keyword" style="color:#00009f">from</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"fast-check"</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">import</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"> assert</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> propertyFor </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">from</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"@fast-check/worker"</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">import</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"> isMainThread </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">from</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"worker_threads"</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">import</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"> pathToFileURL </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">from</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"url"</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> property </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">propertyFor</span><span class="token punctuation" style="color:#393A34">(</span><span class="token function" style="color:#d73a49">pathToFileURL</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">__filename</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> maxAllowedTimeMs </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">10_000</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> maxAllowedTimeMsSpawningWorkerIncluded </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">60_000</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> inputArb </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> todo</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// an arbitrary responsible to generate our inputs</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> fuzzTarget </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">property</span><span class="token punctuation" style="color:#393A34">(</span><span class="token function" style="color:#d73a49">inputArb</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">input</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic">// Arrange</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> startTime </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> performance</span><span class="token punctuation" style="color:#393A34">.</span><span class="token function" style="color:#d73a49">now</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic">// Act</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token function" style="color:#d73a49">functionUnderTest</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic">// Assert</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> endTime </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> performance</span><span class="token punctuation" style="color:#393A34">.</span><span class="token function" style="color:#d73a49">now</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">const</span><span class="token plain"> waitingTimeMs </span><span class="token operator" style="color:#393A34">=</span><span class="token plain"> endTime </span><span class="token operator" style="color:#393A34">-</span><span class="token plain"> startTime</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">if</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">waitingTimeMs </span><span class="token operator" style="color:#393A34">&gt;</span><span class="token plain"> maxAllowedTimeMs</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token keyword" style="color:#00009f">throw</span><span class="token plain"> </span><span class="token keyword" style="color:#00009f">new</span><span class="token plain"> </span><span class="token class-name">Error</span><span class="token punctuation" style="color:#393A34">(</span><span class="token template-string template-punctuation string" style="color:#e3116c">`</span><span class="token template-string string" style="color:#e3116c">Took </span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:#393A34">${</span><span class="token template-string interpolation">waitingTimeMs</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:#393A34">}</span><span class="token template-string string" style="color:#e3116c">ms to run</span><span class="token template-string template-punctuation string" style="color:#e3116c">`</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">if</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">isMainThread</span><span class="token punctuation" style="color:#393A34">)</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  fc</span><span class="token punctuation" style="color:#393A34">.</span><span class="token function" style="color:#d73a49">configureGlobal</span><span class="token punctuation" style="color:#393A34">(</span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    timeout</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> maxAllowedTimeMsSpawningWorkerIncluded</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    endOnFailure</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token boolean" style="color:#36acaa">true</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// we don't want to shrink: we will do that on case by case basis</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    numRuns</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">1_000_000</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// we want more runs than the default 100</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    interruptAfterTimeLimit</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">30</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">*</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">3_600</span><span class="token plain"> </span><span class="token operator" style="color:#393A34">*</span><span class="token plain"> </span><span class="token number" style="color:#36acaa">1_000</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// but don't want to run the fuzzer more than 30 min</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    baseSize</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"xlarge"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic">// we want to run against possibly extra large entries</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token keyword" style="color:#00009f">await</span><span class="token plain"> </span><span class="token function" style="color:#d73a49">assert</span><span class="token punctuation" style="color:#393A34">(</span><span class="token plain">fuzzTarget</span><span class="token punctuation" style="color:#393A34">)</span><span class="token punctuation" style="color:#393A34">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>Our snippet is leveraging two time limits:</p>
<ul>
<li class=""><code>maxAllowedTimeMs</code>: Capture executions leading to unacceptable timings but not able to catch infinite loops.</li>
<li class=""><code>maxAllowedTimeMsSpawningWorkerIncluded</code>: Explicitely higher, it mostly addresses the infinite loop case or any execution running out-of-control for too long. The timing is way higher than the previous one as it also takes into account the time it takes to spawn a worker.</li>
</ul>
<p>Our code being in TypeScript, it has to be transpiled. So we defined the following scripts:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token property" style="color:#36acaa">"build"</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"esbuild ./fuzz.ts --bundle --platform=node --outfile=fuzz.cjs"</span><span class="token punctuation" style="color:#393A34">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token property" style="color:#36acaa">"fuzz"</span><span class="token operator" style="color:#393A34">:</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"node --max_old_space_size=16384 fuzz.cjs"</span><span class="token punctuation" style="color:#393A34">,</span><br></div></code></pre></div></div>
<p>The build script will transform the code starting at <code>./fuzz.ts</code> into a <code>fuzz.cjs</code> able to run by itself in node. The run script will run the test itself.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="takeaways">Takeaways<a href="https://engineering.pigment.com/2024/06/27/fuzz-driven-slow-path-detection/#takeaways" class="hash-link" aria-label="Direct link to Takeaways" title="Direct link to Takeaways" translate="no">​</a></h2>
<p>At first glance, the approach looked too naive to work. But we quickly detected issues as we incrementally plugged new parts of the input in our generator.</p>
<p>A common concern we often heard when we introduced the approach was: inputs might not be realistic and having them realistic enough might be too costly to make the tests useful. Actually, the point is highly challengeable. The aim of this technique is to find bugs that we never thought about so they might seem crazy at first glance. But as the events proved us, there are some unforecasted configurations that would lead us into the red. Instead of waiting for them to come, anticipating them will help us into detecting and patching slow code paths before anything gets reported.</p>
<p>Finally, the fuzzer is the last layer we added in our performance toolbox. We already had many layers to help us in that direction. Contrary to some others it has the benefit to make us able to anticipate issues we never thought about as it will always suggest us with new entries and one day it might detect a performance issue.</p>
<p>For the moment, the execution of the fuzzer has to be manually triggered in the CI. But we plan to run a limited version of it with all our tests: lower number of executions and smaller inputs.</p>]]></content:encoded>
            <category>Performance</category>
            <category>Front</category>
        </item>
        <item>
            <title><![CDATA[Slashing load times by 10x with better scheduling]]></title>
            <link>https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/</link>
            <guid>https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/</guid>
            <pubDate>Mon, 20 May 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Pigment is a super fast growing company. Each day, new clients present unique challenges, adding to the platform's workload and pushing us to continually enhance its performance. This article explains how we successfully divided the loading time for certain clients by up to 10 times without touching to the execution time.]]></description>
            <content:encoded><![CDATA[<p>Pigment is a super fast growing company. Each day, new clients present unique challenges, adding to the platform's workload and pushing us to continually enhance its performance. This article explains how we successfully divided the loading time for certain clients by up to 10 times without touching to the execution time.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="understanding-the-load">Understanding the load<a href="https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/#understanding-the-load" class="hash-link" aria-label="Direct link to Understanding the load" title="Direct link to Understanding the load" translate="no">​</a></h2>
<p>At Pigment, we provide our clients with a solution to transform their raw data into polished reports, financial plans, and real-time visualizations. This empowers them to explore various scenarios, analyze results, and make informed decisions.</p>
<p>However, when dealing with the financial data of major companies like PVH, Figma, Mozilla, Kayak etc. the process of transitioning from data processing to comparing diverse scenarios across numerous perspectives can be extensive. Here's an illustration of a minuscule graph demonstrating the interdependencies among the tasks we need to execute, beginning from data importation:</p>
<p><img decoding="async" loading="lazy" alt="Dependency graph" src="https://engineering.pigment.com/assets/images/dependency-graph-f77c947b7df2b739e8a5601bad0dddb0.png" width="2892" height="1068" class="img_ev3q"></p>
<p>This implies that whenever a user modifies data within a node, we must calculate all subsequent dependencies downstream. What's more challenging is that we enable users to compare various <em>"scenarios"</em>, allowing them to adjust nodes or the original data for comparison, potentially multiplying the number of nodes by the number of scenarios. <strong>A single change could result in thousands of executions 💥</strong>.</p>
<p>When dealing with dozens to hundreds of users for a single client all working at the same time, it's not feasible to manage everything simultaneously. Prioritizing tasks becomes crucial!</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="finding-the-right-prioritization-rules">Finding the right prioritization rules<a href="https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/#finding-the-right-prioritization-rules" class="hash-link" aria-label="Direct link to Finding the right prioritization rules" title="Direct link to Finding the right prioritization rules" translate="no">​</a></h2>
<p>Prioritization algorithms typically fall under the umbrella of scheduling and depend on various external factors: Do we allow preemption? What is the cost of the job? What is the available quota? We won't delve into all of these topics here; they'd deserve their own article! Instead, our focus will be simple: <strong>Do the jobs we execute have significance</strong>?</p>
<p>Initially we assumed that a job matters if a user is observing its outcome. However, this is only partially accurate. A job doesn't exist in isolation; it's interconnected within a broader graph and has dependencies. Therefore, a job matters if it contributes to a result that a user is interested in. For instance, if we assume that our user is only concerned with "Debit card fees" in the graph, we can simplify the graph to:</p>
<p><img decoding="async" loading="lazy" alt="Subgraph" src="https://engineering.pigment.com/assets/images/debit-card-fees-68a6b83a447c93d098d230cb110c0b5b.png" width="2892" height="1086" class="img_ev3q"></p>
<p>Considering this guideline, we conducted a simple test. We selected several clients with a high volume of executions, identified the jobs they were actively monitoring, and determined all the other jobs dependent on those. Then, we categorized the executions into two groups:</p>
<ul>
<li class=""><code>Useful jobs</code>: Those that contributed to a job that someone monitored at least once during the day.</li>
<li class=""><code>Useless jobs</code>: Those that did not.</li>
</ul>
<p>Finally, we calculated a ratio: out of all the jobs awaiting execution, how many useless jobs were being run while there were useful jobs still pending scheduling? Surprisingly, it was found that between 40% to 80% of our executions could be postponed!</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-implementation">The implementation<a href="https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/#the-implementation" class="hash-link" aria-label="Direct link to The implementation" title="Direct link to The implementation" translate="no">​</a></h2>
<p>Before we rolled out this feature, we had to seriously consider how it would handle scalability. We are dealing with potentially hundreds of subscription events pouring in every second, leading to thousands of interconnected jobs in dependency graphs with over 10,000 nodes. The global picture can be sumed up to:</p>
<ol>
<li class="">Identify the jobs impacting what users see.</li>
<li class="">Find all the transitive dependencies leading to those jobs in the dependency graph</li>
<li class="">Arrange those dependencies to guide our scheduler's decision-making.</li>
</ol>
<p><img decoding="async" loading="lazy" alt="Prioritization workflow" src="https://engineering.pigment.com/assets/images/prioritization-workflow-0f83f1b674bff53c3fb845db8dc6c9b4.png" width="2480" height="1380" class="img_ev3q"></p>
<p>Moreover job priorities aren't set in stone. Users frequently switch between dashboards and dependencies in the graph get updated, which means we have to constantly reassess our priorities. We can't react to every change in real-time; we need a system that can handle the load, especially with Pigment's rapid growth each month.</p>
<p>So, we decided to focus on being responsive. After analyzing how users actually use the platform, we noticed they tend to focus on specific parts of their models for short bursts. Subscriptions often target the same set of jobs repeatedly. So, being overly reactive and only focusing on what's happening right now wouldn't be effective. Users move between dashboards, and we can't slow down computations for jobs they've temporarily left behind.</p>
<p>So what we did is implementating our prioritization based on a sliding window. Everytime we receive a subscription, we just update the time at which the targeted jobs were last monitored. If it newly enters the sliding window, it means that we may need to prioritize it and the prioritization service will get notified. If not, we won't bother trying.</p>
<p><img decoding="async" loading="lazy" alt="Sliding window" src="https://engineering.pigment.com/assets/images/sliding-window-56994c192035c218c39747bfd2766ab2.png" width="2172" height="615" class="img_ev3q"></p>
<p>The cleanup process will be handled by an asynchronous cron job. Periodically, it will gather all jobs within the sliding window and perform a complete recalculation of priorities. This allows us to automatically incorporate the latest version of the dependency graph and clean up both dependencies that are no longer accessible and dependencies of jobs that are no longer monitored.</p>
<p>This approach offers the advantage of being highly responsive to new jobs while significantly reducing the overall workload. However, it comes with the potential drawback of prioritizing jobs that are no longer actively monitored by our users. Nevertheless, if we find ourselves inundated with irrelevant jobs, we can always adjust by reducing the sliding window duration and increasing the frequency of the cron job.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="final-results">Final results<a href="https://engineering.pigment.com/2024/05/20/job-prioritization-at-pigment/#final-results" class="hash-link" aria-label="Direct link to Final results" title="Direct link to Final results" translate="no">​</a></h2>
<p>Ultimately, this incredibly straightforward prioritization method enabled a reduction in the loading time of certain highly viewed segments for certain clients by up to 10! The numerous scenarios and applications that were not currently in use would no longer delay crucial executions, we consistently executed only the most efficient pathway to update what they were viewing.</p>]]></content:encoded>
            <category>Performance</category>
            <category>Scheduling</category>
        </item>
        <item>
            <title><![CDATA[Inside Pigment: Optimizing Infrastructure Costs with Our Own Tools]]></title>
            <link>https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/</link>
            <guid>https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/</guid>
            <pubDate>Mon, 13 May 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Pigment is not even 5 years old, yet our R&D team has reached a FinOps maturity stage we are already proud of. Thanks to our own product, we enable cross functional use cases that serve finance, sales, and tech teams throughout our customer lifecycle.]]></description>
            <content:encoded><![CDATA[<p>Pigment is not even 5 years old, yet our R&amp;D team has reached a FinOps maturity stage we are already proud of. Thanks to our own product, we enable cross functional use cases that serve finance, sales, and tech teams throughout our customer lifecycle.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="finops-pigment-rd">FinOps @Pigment R&amp;D<a href="https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/#finops-pigment-rd" class="hash-link" aria-label="Direct link to FinOps @Pigment R&amp;D" title="Direct link to FinOps @Pigment R&amp;D" translate="no">​</a></h2>
<p>Today, a Pigment application fully calculates the lines of our infrastructure invoice automatically, covering our cloud provider and third party software bills. As a FinOps team, we reconcile this data with the official invoice at the end of every month, and drill down into necessary details, down to the day and the SKU level. We also monitor the cost of every individual Pigment customer, over time and based on their platform usage. Finally, we follow-up on operational KPIs together with our engineers so we detect and fix bad trends, and monitor whether we need to renew cloud commitments.</p>
<p><em>We do this entirely in our own product, Pigment, supported by custom developments made by our Site Reliability Engineering team.</em></p>
<figure><p><img decoding="async" loading="lazy" alt="Finops Article Illustration 1" src="https://engineering.pigment.com/assets/images/finops_article_1-dc7deb996ead4709a031185344bb4a38.png" width="1600" height="835" class="img_ev3q">
</p><figcaption><em>Our FinOps effort started in July, 2023. Immediately after, our costs got under control.</em></figcaption><p></p></figure>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-it-started">How it started<a href="https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/#how-it-started" class="hash-link" aria-label="Direct link to How it started" title="Direct link to How it started" translate="no">​</a></h2>
<p>Does such an advanced model really make sense at this stage? Indeed, a startup has to grow fast. To release and scale the product as fast as possible, our engineers need to spawn environments, to log traces, to size up our infra, in an almost limitless fashion. As they do so, many things can be overlooked. It may be technical debt, clean processes, but also, the cash we spend. Yet, funding has gone through a major crisis lately. Raising more money when it runs out is no longer an option at most companies. Growth needs to be controlled, more than was necessary perhaps 2 years ago.</p>
<p>How do we plan and achieve a reasonable level of control in an engineering team whose ability to work well and fast is at the heart of the company's success? At Pigment, we are never driven by “doing things by the book”. We are constantly driven by pragmatism, and pragmatism relies on framing the right problems to solve.</p>
<p>Indeed, what our management asked our R&amp;D team wasn’t initially to get better at monitoring our costs. It was rather about understanding what a specific customer implementation costs the company. This precious knowledge has been the starting point to help us solve multiple problems at once.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-problems-did-we-need-to-solve">What problems did we need to solve?<a href="https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/#what-problems-did-we-need-to-solve" class="hash-link" aria-label="Direct link to What problems did we need to solve?" title="Direct link to What problems did we need to solve?" translate="no">​</a></h2>
<p><strong>Firstly, calculate profitability.</strong> Comparing infra cost and direct support cost to annual recurring revenue of a given customer, we can then estimate how profitable its current implementation is. Technical or customer success teams can then take multiple actions to optimize this profitability, such as reviewing a client’s data model, or analyzing whether their usage is still cost efficient.</p>
<figure><p><img decoding="async" loading="lazy" alt="Finops Article Illustration 2" src="https://engineering.pigment.com/assets/images/finops_article_2-1c93093059b5a751d10c1a52e084627a.png" width="1600" height="671" class="img_ev3q">
</p><figcaption><em>Understanding the profitability of clients (this is not real data)</em></figcaption><p></p></figure>
<p><strong>Secondly, determine pricing.</strong> To estimate the cost of a given client’s infrastructure, once it is live with Pigment, our FinOps team looks at different data points. For instance, we determine allocation keys to attribute the cost of mutualized resources to individual customers, based on their platform usage. Could we reuse these data points to scope new deals, that is, to find a pricing that meets our client’s interests while protecting our margin? The more data we have, the better our knowledge: our FinOps team can support our business teams to price deals accordingly, early on in our company’s life.</p>
<p><strong>Finally, support forecasting.</strong> There are 3 questions our FinOps team needs to answer on a regular basis. These questions are like Russian dolls:</p>
<ol>
<li class="">
<p><em>What we think we’ll do</em>: this is our official yearly budget. In the first iterations, it will be very close to linearly scaling costs based on the forecasted business growth and adding a lot of contingency on top of it. But the goal is to get this forecast closer to our profitability targets over time.</p>
</li>
<li class="">
<p><em>What we hope we’ll do</em>: this is the operational target we can set to engineering teams. While we may have planned contingencies at a financial level, we can be more ambitious by setting targets and reaping good news for our finance teams throughout the year, as we optimize the usage of our infrastructure. Hopefully, we don’t have as much contingency to add with every budgeting cycle.</p>
</li>
<li class="">
<p><em>What we say we’ll do</em>: this is the spend we are ready to commit on to third parties, like cloud providers, in exchange for commercial returns like discounts. Our commitment level is a risk : it needs to be carefully assessed while preserving our partnerships.</p>
</li>
</ol>
<figure><p><img decoding="async" loading="lazy" alt="Finops Article Illustration 3" src="https://engineering.pigment.com/assets/images/finops_article_3-c665db09b5d7bfc6f85f813a87c9c1bc.png" width="1600" height="345" class="img_ev3q">
</p><figcaption><em>A view of cost &amp; savings on a data sample</em></figcaption><p></p></figure>
<div class="theme-admonition theme-admonition-info admonition_xJq3 alert alert--info"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Allocating the cost of mutualized resources: how we do it</div><div class="admonitionContent_BuS1"><p>When allocating the cost of mutualized resources (like a shared database) between customers, a first and reasonable strategy is to allocate it uniformly between the users of this resource: it is quick to implement, works well if you have the same usage from most customers.</p><p>The evident drawback is that if some clients have a much bigger usage of the resource than others, this allocation strategy will not reflect it.</p><p>At Pigment, database costs are a significant part of our cloud costs. And clients have vastly different database workloads. This motivated us in building a more ambitious FinOps model to better track the cloud cost of each client.</p><p>To do so, nothing better than talking to the persons building Pigment, the developers, and the infrastructure team, in order to find a reasonable heuristic to allocate costs of shared resources.</p><p>The Pigment solution is many things, but from a low-level point of view, it executes jobs to process data.</p><p>When a job is completed, we retrieve its type, duration, the client and database host and export this data to BigQuery. Once this data is available in BigQuery, we can query it to estimate the ratio “client usage time” versus the “total usage time” of a database host: this ratio is used to allocate the cost of shared resources.</p><p>Does it allow us to allocate the cost from each client to the cent? No. Do we care? Still no!</p><p>It is a good enough approximation, giving us the precision we need to understand our cloud costs, without building an over-engineered solution that we will have to maintain.</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="focus-on-forecasting-a-cross-functional-effort">Focus on forecasting: a cross functional effort<a href="https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/#focus-on-forecasting-a-cross-functional-effort" class="hash-link" aria-label="Direct link to Focus on forecasting: a cross functional effort" title="Direct link to Focus on forecasting: a cross functional effort" translate="no">​</a></h2>
<p>When we established our first baseline forecast, we realized only 40% of our total cost was truly influenced by the growth of our business. The rest was either rather constant, or scaling with other factors like our internal growth, such as our software engineers growing headcount. It was thus relatively easy to define targets for our engineering teams on the latter, for instance, to set a budget for our test environments.</p>
<p>On the other hand, when the cost is not easily predictable, typically, incurred by our customer’s usage and new deals coming up, then we can use different methods. We can apply a month-over-month growth as a baseline. But since we now have a cost per customer, along with other business metrics, we can work on more detailed scenarios.</p>
<p><em>Defining those scenarios, and revising them over time, is a cross-functional effort.</em></p>
<p>Engineering teams can complete theoretical numbers with a reality check. For instance:</p>
<ul>
<li class="">mutualize resources whenever possible so our infrastructure doesn’t grow linearly with the number of customers,</li>
<li class="">use different storage classes so related costs do not explode with the volumes we store,</li>
<li class="">limit how observability costs grow with traffic by leveraging log levels more. We once saved 10% of our Datadog costs with a single PR!</li>
</ul>
<p>Defining related projects early on, even if we’ll execute them later in the year, helps us set operational targets and negotiate commercial agreements, while estimating a better contingency with finance.</p>
<p>With increased understanding of the relationship between business metrics and infra costs, customer facing teams take part in this too. For instance, pre-sales teams can add new questions in their deal assessment templates, so they gather from our prospects the business metrics that most influence cost. Should we put this data in Pigment, there would be a fully connected process from pricing to budgeting, and a feedback loop to revise their relationship over time.</p>
<p>Finally, we leverage commercial agreements to make our cost more predictable. It is much more profitable at some point to negotiate EULAs (End-User License Agreements) with third parties to cap our total cost rather than let it grow linearly along our business.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="using-pigment-a-no-brainer">Using Pigment: a no brainer<a href="https://engineering.pigment.com/2024/05/13/optimizing-infrastructure-costs/#using-pigment-a-no-brainer" class="hash-link" aria-label="Direct link to Using Pigment: a no brainer" title="Direct link to Using Pigment: a no brainer" translate="no">​</a></h2>
<p>The initial work on customer profitability could have been done with a spreadsheet and a couple of meetings, but it would have generated more work to repeat that exercise in the future. Moreover, it would have remained a work only a few people could do. At Pigment, we have scale at heart, and access to the most powerful FP&amp;A solution on the market! For us as engineers, it was clear the data we worked on had to be published through repeatable processes, and made available in self service to the rest of the company, on a need-to-know basis. <em>Building this entirely in the Pigment product was a no-brainer</em>, especially as other teams made the same move, opening up opportunities to connect data together.</p>
<p>Today, our FinOps team reviews costs on a monthly basis with our Finance team. They used to contact our SRE lead directly in Slack DMs to get the information their job required! Thanks to the reports we built, they are now autonomous in reconciling invoices and fetching accounting data. <em>The responsibility shifts to the left to the organization</em>, preserving the focus of our engineers.</p>
<p>We also share those costs with the rest of the R&amp;D team on a regular basis. When a team is accountable, being transparent only yields benefits. Our engineers have not changed their way of work : we don’t want to hurt our operational efficiency. But because our numbers are regularly made public, any software engineer can understand the impact of what they ship on the cost we are billed. Doing so generates engagement almost for free, already triggering numerous initiatives.</p>
<p>On top of everything, this is a unique opportunity for anyone in the R&amp;D team to relate to our customers’ use cases, and realize the incredible potential of our product!</p>]]></content:encoded>
            <category>FinOps</category>
            <category>Infrastructure</category>
        </item>
        <item>
            <title><![CDATA[Pure Terraform modules]]></title>
            <link>https://engineering.pigment.com/2024/04/03/pure-terraform-modules/</link>
            <guid>https://engineering.pigment.com/2024/04/03/pure-terraform-modules/</guid>
            <pubDate>Wed, 03 Apr 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[As our infrastructure scales, our Terraform codebase grows. To remain]]></description>
            <content:encoded><![CDATA[<p>As our infrastructure scales, our Terraform codebase grows. To remain
productive, we need to keep our code flexible and extendable. A design pattern
we find useful is the pure Terraform module. Let's dive into what it is and how
it helps us write maintainable Terraform code.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-are-pure-modules">What are pure modules?<a href="https://engineering.pigment.com/2024/04/03/pure-terraform-modules/#what-are-pure-modules" class="hash-link" aria-label="Direct link to What are pure modules?" title="Direct link to What are pure modules?" translate="no">​</a></h2>
<p>We are borrowing the term "pure" from functional programming. In functional
programming, a pure function is a function that does not have any side effects.
It does not mutate any state outside of its scope. Pure functions are easier to
reason about and test. They are also easier to reuse.</p>
<p>Just like pure functions, pure Terraform modules do not interact with any
external system. They do not create any resources, read any data sources, or use
any providers. Their role is to provide information about our infrastructure to
the caller. This seems counterintuitive at first. After all, Terraform is all
about creating and managing resources. However, we find that pure modules are a
useful way to structure our Terraform codebase.</p>
<p>Let's look at an example from our codebase: the <code>cluster_info</code> module. It
provides information about our Kubernetes clusters. We call it like this:</p>
<div class="language-hcl codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-hcl codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">module</span><span class="token keyword type variable" style="color:#36acaa"> "info" </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">source</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"../cluster_info"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">cluster_id</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"production-eu1"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>The <code>module.info</code> object now contains information about our cluster that we can
use to create resources, configure providers, or express business logic. Once
called, the module is essentially an object:</p>
<div class="language-hcl codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-hcl codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token property" style="color:#36acaa">module.info</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">name</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"production"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">purpose</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"product"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">project</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"pigment-production-123456"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">region</span><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west3"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">cidr</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.12.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># and more...</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>The <code>cluster_info</code> module's implementation is straightforward. It contains
information about our clusters, organized by cluster ID. When called, it looks
up the information for the given cluster ID and returns it:</p>
<div class="language-hcl codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-hcl codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">variable</span><span class="token keyword type variable" style="color:#36acaa"> "cluster_id" </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">description</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"ID of the cluster to provide info about"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">type</span><span class="token plain">        </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> string</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">locals</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># The GCP region where each cluster is located.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">region_by_cluster</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">argocd</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west3"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Frankfurt, DE</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-eu1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west3"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Frankfurt, DE</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-eu1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west9"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Paris, FR</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-us1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"us-west1"</span><span class="token plain">    </span><span class="token comment" style="color:#999988;font-style:italic"># Oregon, US</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-us1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"us-central1"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Iowa, US</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-vault-eu1</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west9"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Paris, FR</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-vault-eu2</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west1"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># St. Ghislain, BE</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-eu1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west3"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Frankfurt, DE</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-eu1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"europe-west9"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Paris, FR</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-us1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"us-west1"</span><span class="token plain">    </span><span class="token comment" style="color:#999988;font-style:italic"># Oregon, US</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-us1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"us-central1"</span><span class="token plain"> </span><span class="token comment" style="color:#999988;font-style:italic"># Iowa, US</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">region</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> local.region_by_cluster</span><span class="token punctuation" style="color:#393A34">[</span><span class="token plain">var.cluster_id</span><span class="token punctuation" style="color:#393A34">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">output</span><span class="token keyword type variable" style="color:#36acaa"> "region" </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">description</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"GCP region where the cluster is located"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">value</span><span class="token plain">       </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> local.region</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>Beyond simple lookups, a pure module can also contain logic to generate the
information it provides. For example, the naming convention for our clusters
changed as their number grew. The <code>cluster_info</code> module's implementation
reflects that:</p>
<div class="language-hcl codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-hcl codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token keyword" style="color:#00009f">locals</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># Our naming conventions changed when Pigment became multi-regional.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># Newer clusters use their ID as their name.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># Older clusters' names are tracked here.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">default_name</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> var.cluster_id</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">name_override_by_cluster</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-eu1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"staging"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-eu1</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"production"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">name</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> lookup(local.name_by_cluster, var.cluster_id, local.default_name)</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token keyword" style="color:#00009f">output</span><span class="token keyword type variable" style="color:#36acaa"> "name" </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">description</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"Name of the cluster"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">value</span><span class="token plain">       </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> local.name</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain"></span><span class="token punctuation" style="color:#393A34">}</span><br></div></code></pre></div></div>
<p>We find that grouping information about our infrastructure in this way fits well
with how we use Terraform.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="key-benefits-of-pure-modules">Key benefits of pure modules<a href="https://engineering.pigment.com/2024/04/03/pure-terraform-modules/#key-benefits-of-pure-modules" class="hash-link" aria-label="Direct link to Key benefits of pure modules" title="Direct link to Key benefits of pure modules" translate="no">​</a></h2>
<p>Pure modules have no effect on the provisioned infrastructure. They're sole
impact is on how we write and structure our Terraform code. Their extreme
lightness makes them pretty much free to call. Any part of our codebase can call
pure modules without worrying about side effects. There is no need to pass
information around as variables or duplicate it, since we can look up the
information we need wherever we need it.</p>
<p>Using pure modules reduces the need for reading remote states. Using
<code>terraform_remote_state</code> data sources is a common pattern in Terraform, but it
comes at a cost. It creates a strong coupling between different parts of our
codebase. Pure modules allow us to avoid this coupling when possible. Any static
information about our infrastructure can be stored in a pure module. We only
need to read remote states when we need dynamic information, like the ID of a
resource that was created in a separate part of our infrastructure. This makes
our codebase much more modular and reduces the need for complex data flows.</p>
<p>Pure modules are also easy to extend as our infrastructure grows. Adding new
information to them does not affect the rest of the codebase. For instance,
adding a new cluster to our infrastructure is as simple as adding a new entry to
the relevant maps in our pure modules.</p>
<p>Pure modules can also implement simple logic to handle edge cases. We found that
this allows us to compartmentalize the complexity of our infrastructure and keep
our codebase clean. For instance, modules can look up a cluster's name without
worrying about how our naming conventions have changed over time.</p>
<p>Additionally, pure modules centralize core information about our infrastructure.
Rather than having to look up information in multiple places, we can find it all
in one place. This makes it easier to reason about our infrastructure and allows
our code to act as an inventory of our infrastructure. A good example of this is
network IP ranges. Our codebase contains a file called <code>cluster_network_info.tf</code>
that contains all the IP ranges for our clusters. This makes it easy to
understand our network layout and to avoid IP conflicts:</p>
<div class="language-hcl codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-hcl codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token comment" style="color:#999988;font-style:italic"># The IP range used by k8s pods, services, and nodes for a given cluster.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">k8s_cidr_by_cluster</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">argocd</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.201.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-eu1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.101.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-eu1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.111.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-us1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.102.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-us1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.112.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-vault-eu1</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.103.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">staging-vault-eu2</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.104.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-eu1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.1.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-eu1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.11.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-us1</span><span class="token plain">    </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.2.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">    </span><span class="token property" style="color:#36acaa">production-us1-dr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> </span><span class="token string" style="color:#e3116c">"10.12.0.0/16"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token punctuation" style="color:#393A34">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">  </span><span class="token property" style="color:#36acaa">k8s_cidr</span><span class="token plain"> </span><span class="token punctuation" style="color:#393A34">=</span><span class="token plain"> local.k8s_cidr_by_cluster</span><span class="token punctuation" style="color:#393A34">[</span><span class="token plain">var.cluster_id</span><span class="token punctuation" style="color:#393A34">]</span><br></div></code></pre></div></div>
<p>Finally, pure modules are fully compatible with vanilla Terraform. There is no
need for any special tooling or plugins, no need to generate code or orchestrate
Terraform commands, and no need to ramp up on more technology. Pure modules are
just Terraform modules that provide information about our infrastructure.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="takeaways">Takeaways<a href="https://engineering.pigment.com/2024/04/03/pure-terraform-modules/#takeaways" class="hash-link" aria-label="Direct link to Takeaways" title="Direct link to Takeaways" translate="no">​</a></h2>
<p>Terraform HCL, as a declarative language, can benefit from patterns from
functional programming. Using tried and true software design patterns is
essential to keeping a Terraform codebase modular and easy to understand,
which we need to scale our infrastructure without scaling our technical debt.</p>
<p>Pure modules are a useful way to structure Terraform code. They provide a way
to keep complexity contained and to centralize information about our resources.</p>
<p>In our experience, pure modules have helped us keep our Terraform codebase
clean, maintainable, and scalable. We hope that you find them useful in your
Terraform projects as well.</p>]]></content:encoded>
            <category>Software Design</category>
            <category>Terraform</category>
            <category>Infrastructure</category>
        </item>
        <item>
            <title><![CDATA[The journey towards Tech Blog]]></title>
            <link>https://engineering.pigment.com/2024/04/02/the-journey-towards-tech-blog/</link>
            <guid>https://engineering.pigment.com/2024/04/02/the-journey-towards-tech-blog/</guid>
            <pubDate>Tue, 02 Apr 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Many mature companies publish their own technical blog. But have you ever wondered how to launch such an initiative as your company grows, when to do it, and how to make it real? This post describes how we spawned our first blog at Pigment and the various challenges and questions we addressed to reach that goal.]]></description>
            <content:encoded><![CDATA[<p>Many mature companies publish their own technical blog. But have you ever wondered how to launch such an initiative as your company grows, when to do it, and how to make it real? This post describes how we spawned our first blog at Pigment and the various challenges and questions we addressed to reach that goal.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="blog-or-no-blog">Blog or no blog?<a href="https://engineering.pigment.com/2024/04/02/the-journey-towards-tech-blog/#blog-or-no-blog" class="hash-link" aria-label="Direct link to Blog or no blog?" title="Direct link to Blog or no blog?" translate="no">​</a></h2>
<p>Technical blogs and articles are often a way for companies to communicate on the technical challenges they had and have to overcome. They tend to materialize for external developers what the company is doing from a technical point-of-view on a daily basis. While it can be considered as a powerful asset to better advertise on the strengths of the teams, it tends to require a bunch of time and energy. Indeed, a blog does not live by itself as it needs new content and thus time. It also needs people that like to share what they learnt, discovered, or faced.</p>
<p>When you are in a start-up with 4 developers, setting up a blog is definitely not your first target. You and your team are aiming to make the difference with competing products. This is especially true when your product does not target a developer audience, as a blog may not seem like a viable option for encouraging adoption and spreading the word about your tool. As such the priority starts low but at some point, it may become a clear will that you as a company or as an individual want to share on what you achieved and how you overcame part of these challenges. Question being to find this point in time.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="fostering-a-sharing-culture">Fostering a sharing culture<a href="https://engineering.pigment.com/2024/04/02/the-journey-towards-tech-blog/#fostering-a-sharing-culture" class="hash-link" aria-label="Direct link to Fostering a sharing culture" title="Direct link to Fostering a sharing culture" translate="no">​</a></h2>
<p>A blog is mostly about sharing. It is also about getting advice, suggestions, or questions from the external world, on simple to complex challenges.</p>
<p>At Pigment, sharing across R&amp;D has always been one of our key principles. From the start, our development process has integrated the concept of Request For Comments (RFC). This practice is guiding us before we begin any new feature. RFCs enable us to closely plan each feature. We consider the technical requirements, product impacts, and design aspects. This approach ensures a well-rounded development process.</p>
<p>These documents in addition to being the backbone for any new feature to succeed, are a way to share knowledge into our R&amp;D. For instance, if someone wants to understand why we did something in a given way, there is a document explaining the pros and cons of the alternatives we thought about in the past. Documents have always been written to ensure accessibility for everyone, including remote workers, who have been a significant part of our team from day one. They have also always been shared internally to ensure consensus and address any remaining blockers.</p>
<p>We also quickly fostered sharing extending outside of the R&amp;D for new features to be demo-ed and we encouraged technical exchanges among developers, allowing them to showcase new discoveries on various technical topics.</p>
<p>But we wanted to push it further with an external communication layer made of a technical blog, this blog.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="tech-stack">Tech stack<a href="https://engineering.pigment.com/2024/04/02/the-journey-towards-tech-blog/#tech-stack" class="hash-link" aria-label="Direct link to Tech stack" title="Direct link to Tech stack" translate="no">​</a></h2>
<p>As our culture of sharing and openness took root, it became clear that we needed a blog able to reflect these values. Our next challenge was to choose the right technical stack for it. Indeed, there are many ways to publish a blog: from on-premise well-known blogging platforms to home-made options.</p>
<p>At Pigment, our blog serves two main purposes. First, it acts as a window, offering the external world a glimpse into our daily work. Second, it mirrors our core principles and objectives. We aim for our blog to embody the values and goals that guide us. As such our blog must be a way to show:</p>
<ul>
<li class="">live demos of components,</li>
<li class="">designs we worked on,</li>
<li class="">code snippets,</li>
<li class="">playgrounds…</li>
</ul>
<p>We decided to put aside external platforms, in order to keep those capabilities. We embraced a JavaScript and React based option making us able to share and customize more easily our content. Our choice went for <a href="https://docusaurus.io/" target="_blank" rel="noopener noreferrer" class="">Docusaurus</a>. As a blog is also about publishing it, we wanted to make the publication workflow as simple as possible. For our blog's deployment, we chose an automated process. This process uses <a href="https://docs.github.com/fr/actions" target="_blank" rel="noopener noreferrer" class="">GitHub Actions</a>, a tool for automation. With GitHub Actions, our content gets automatically published to <a href="https://pages.github.com/" target="_blank" rel="noopener noreferrer" class="">GitHub Pages</a>.</p>]]></content:encoded>
            <category>Return of Experience</category>
        </item>
    </channel>
</rss>