[{"data":1,"prerenderedAt":398},["ShallowReactive",2],{"e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5":3,"lateste0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5":49,"jobs-count":397},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"date":8,"id":9,"collection":5,"content":10,"coverImage":7,"author":22,"categories":23,"card":28,"order":22,"slug":35,"title":29,"uri":34,"url":34,"meta":36,"canonical":22,"social":39,"_id":44,"_type":45,"_source":46,"_file":47,"_stem":48,"_extension":45},"/tech/e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5","tech",false,"","2024-07-28T22:00:00.000000Z","e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5",[11,14,20],{"type":12,"text":13},"text","\u003Cp>Hello, world! That line’s been printed to console by every programmer at least once in their lifetime. And it’s also an appropriate way to welcome you all to the Outfit7 Tech Blog.\u003C/p>\u003Cp>Outfit7? We’re the people behind \u003Ca target=\"_blank\" href=\"https://talkingtomandfriends.com/\">Talking Tom & Friends\u003C/a> (which you’ve probably heard of). The brand was launched in 2010 and currently has a portfolio of 20+ games (\u003Ca target=\"_blank\" href=\"https://o7n.co/mta2-o7web-games-news\">My Talking Angela 2\u003C/a>, \u003Ca target=\"_blank\" href=\"https://o7n.co/mttf-o7web-games-news\">My Talking Tom Friends\u003C/a>, \u003Ca target=\"_blank\" href=\"https://o7n.co/mtt-o7web-games-news\">My Talking Tom\u003C/a>, \u003Ca target=\"_blank\" href=\"https://o7n.co/ttgr-o7web-games-news\">Talking Tom Gold Run\u003C/a>, etc.) that have been downloaded more than 19 billion times! In fact, Talking Tom & Friends has been the most downloaded mobile game franchise worldwide for 10 years now (2013–2022).\u003C/p>\u003Cp>But we all know that humongous download numbers are only one part of the story. Having the tech to handle 500 million monthly users, 5TB of data per day, and thousands of multiplayer users across the globe simultaneously is not a walk in the park. Unless, of course, you take that walk with a hundred brilliant tech experts (and you still walk for a while). And that’s what we did (metaphor aside).\u003C/p>\u003Cp>Over the past 10 years, a lot has changed. We no longer fit on one floor of our office building (actually we no longer fit on eight floors). Our tech department went from four people to over 120. Our one wannabe analytics employee became a department of over 10 professionals. And we’ve now automated more tasks and written more unit, integration, and e2e tests than there are monkey NFTs. And, you know, we released a couple of games, each with billions of downloads.\u003C/p>\u003Cp>But some stuff has (amazingly) not changed.\u003C/p>\u003Cp>We’re gamers and we love technology in all shapes and sizes. We’ve always strived to use all the latest technologies that the mighty tech world has to offer, even though that meant rewriting/refactoring/restructuring our code again and again. We switched our game engine a couple of times and even wrote our own game engine (that was a tough one). We went from old Java’s JSP dashboards to AngularJS, and from there to Angular and now to Vue.js. And we’ve tried all the different game architectures — lately the ECS.\u003C/p>",{"type":15,"items":16},"slider",[17],{"caption":18,"asset":19},"A simplified view of how we connect different parts of our games, from Game to Native and Backend. Every part is developed by one or multiple teams — connecting all these teams and departments is a technical challenge of its own.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/the-outfit7-blog-ibi.jpg",{"type":12,"text":21},"\u003Cp>\u003Ca href=\"https://outfit7.com/careers/jobs?field=technology\">\u003Cstrong>Technology\u003C/strong>\u003C/a>\u003Cstrong> is at our core here at Outfit7.\u003C/strong> We try everything that we think might be either beneficial or even just interesting to do. Some stuff fails miserably, but some results are just amazing. And we’d like to share what we’ve learned with you. It’ll give you some insights into how we do stuff. And maybe it’ll spare you a couple of months of investigation or give you some ideas of what else you could try optimizing, refactoring, automating or taking to the next level.\u003C/p>\u003Cp>Our Outfit7 experts — people of over 25 nationalities from all of our technology departments (Game, Android, iOS, Backend, DevOps, Web, and QA) — are really looking forward to sharing their stories, experiences, and expertise on the Outfit7 Tech Blog, both for the mentioned and not-yet-mentioned topics. This blog is our chance to engage with the tech world and provide you with quality and original content. We hope you find it interesting and useful.\u003C/p>",null,[24],{"id":25,"title":26,"slug":27},"tech_categories::web","Web","web",{"title":29,"copy":22,"backgroundImage":30,"cta":31},"The Outfit7 Tech Blog","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/the-o7-tech-blog-card.jpg",{"title":32,"type":33,"href":34},"Read the tech article","link","/blog/tech/the-outfit7-tech-blog","the-outfit7-tech-blog",{"title":37,"description":7,"canonical":38,"robots":7},"The Outfit7 Tech Blog | Outfit7","http://localhost/blog/tech/the-outfit7-tech-blog",{"open_graph":40,"twitter":40,"site_name":41,"title":29,"description":42,"image":43},true,"Outfit7","Makers of Talking Tom & Friends – Fun Games for All Ages","https://cdn-o7.o7web.com/img/asset/YXNzZXRzLzRfYmxvZ19uZXdzL2xhdGVzdF9uZXdzL291dGZpdDctbmV4dC1nZW4tZHNjMDg3NzYuanBn?w=1200&h=630&q=70&fit=crop&s=bb4de7e9c4321ba0195125b9a0a59bc6","content:tech:e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5.json","json","content","tech/e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5.json","tech/e0b0ee2b-ae02-4de6-bf4f-d06e2c4159e5",[50,255,321],{"_path":51,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"date":52,"id":53,"collection":5,"content":54,"coverImage":7,"author":239,"categories":240,"card":241,"order":22,"slug":246,"title":242,"uri":245,"url":245,"meta":247,"canonical":22,"social":250,"_id":252,"_type":45,"_source":46,"_file":253,"_stem":254,"_extension":45},"/tech/159f8d76-b27c-4cf1-8655-a8faa65f15cf","2025-06-18T22:00:00.000000Z","159f8d76-b27c-4cf1-8655-a8faa65f15cf",[55,57,62,64,87,89,94,96,101,103,108,110,115,117,121,123,127,129,136,138,143,145,149,151,155,157,161,163,168,170,175,177,191,193,198,200,208,210,215,217,221,223,230,232,237],{"type":12,"text":56},"\u003Cp>Our newly updated game, My Talking Hank: Islands, is set on a tropical island inhabited by a group of fun and friendly animals, each of which has different interests and activities. When brainstorming possible activities, we decided to develop something around a hair salon, where players could groom a lion’s mane.\u003C/p>",{"type":15,"items":58},[59],{"caption":60,"asset":61},"King of the jungle","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image7.png",{"type":12,"text":63},"\u003Cp>Before we get into the details of the project, let me introduce myself. \u003C/p>\u003Cp>My name is Marko Ravnjak and I’ve been with Outfit7 since 2015. I work as a Senior Software Engineer, and the scope of my work ranges across all the programming aspects of game development, from making tools and editors to writing libraries and core systems, to actual game logic and features. So, let’s get down to it!\u003C/p>\u003Cp>The concept of this feature was divided into a few steps: \u003C/p>",{"type":15,"items":65},[66,69,72,75,78,81,84],{"caption":67,"asset":68},"The player sees a messy lion.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image12.png",{"caption":70,"asset":71},"The lion is automatically washed.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image2.png",{"caption":73,"asset":74},"A player uses a fan to dry the mane.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image5.png",{"caption":76,"asset":77},"Different tools can be used to groom and style the hair.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image8.png",{"caption":79,"asset":80},"Hair spray and color can be applied.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image24.png",{"caption":82,"asset":83},"The player can add accessories.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image25.png",{"caption":85,"asset":86},"And finally, we celebrate the look.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image11.png",{"type":12,"text":88},"\u003Cp>My Talking Hank: Islands was being developed with Unity, so I tried to use/reuse components and systems as much as possible. The basic idea was to have a front-facing lion with hair strands covering its head. Each strand would be a 2D object, also oriented towards the front. For such objects, a line renderer is the optimal choice. A line renderer is a dynamic mesh, generated from a set of positions. Together with line width, each position defines a segment of the line so that you get a continuous strip of triangles. \u003C/p>",{"type":15,"items":90},[91],{"caption":92,"asset":93},"Line renderer (shaded wireframe mode).","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image22.png",{"type":12,"text":95},"\u003Cp>The tricky part with line renderers is the curvature, where segments (if wide enough) can overlap, which the line mesh generator must detect and fix. Fortunately, Unity provides a decent \u003Ca href=\"https://docs.unity3d.com/Manual/class-LineRenderer.html\">\u003Cu>LineRenderer\u003C/u>\u003C/a> component (it is lacking in one or two areas, but more on that later).\u003C/p>",{"type":15,"items":97},[98],{"caption":99,"asset":100},"Exaggerated angle break.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image15.png",{"type":12,"text":102},"\u003Cp>Next, we needed a system that could “drive” these line renderers and allow us to modify the mesh according to our wishes and available tools. We decided to make the hair strands somewhat static, meaning that users can move or drag them around as they wish, but once they stop, the strands stay still (i.e. they don’t fall down). This would allow users to design different hair cuts/shapes without losing their progress, as it were.\u003C/p>\u003Cp>The hair strands also needed to be flexible. They needed to be able to bend without breaking. The obvious solution here would be a \u003Ca href=\"https://www.google.com/search?q=what+are+splines+in+computer+graphics&rlz=1C1GCCA_en&oq=what+are+splines+in+computer+gra&gs_lcrp=EgZjaHJvbWUqCAgBEAAYFhgeMgYIABBFGDkyCAgBEAAYFhgeMggIAhAAGBYYHjIICAMQABgWGB4yCAgEEAAYFhgeMgoIBRAAGA8YFhgeMg0IBhAAGIYDGIAEGIoFMg0IBxAAGIYDGIAEGIoFMg0ICBAAGIYDGIAEGIoFMg0ICRAAGIYDGIAEGIoF0gEINjY0MGowajSoAgCwAgA&sourceid=chrome&ie=UTF-8\">\u003Cu>spline system\u003C/u>\u003C/a>, which is a set of control points or so-called “knots” (a position, tangents can also be added) from which smooth, curved lines can be generated.\u003C/p>\u003Cp>The first option I tried was \u003Ca href=\"https://docs.unity3d.com/Packages/com.unity.splines@2.4/manual/index.html\">\u003Cu>Unity’s Spline Package\u003C/u>\u003C/a>. I’d used this on a previous project and it had worked well, offering great editing tools while being easy to use. \u003C/p>",{"type":15,"items":104},[105],{"caption":106,"asset":107},"Unity’s Splines system","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image29.png",{"type":12,"text":109},"\u003Cp>But, as it turned out, the Unity Splines Package wasn’t really meant for runtime editing. We’d previously used it to define a path across a level (editor time) where a player was moving during gameplay. In this instance, the spline wasn’t being changed during runtime, only evaluated (meaning getting the position and tangent at at a given time). Editing the spline data and rebuilding it in runtime turned out to be inefficient (mostly memory-wise - GC) for the My Talking Hank: Islands project.\u003C/p>\u003Cp>I reverted to the good old \u003Ca href=\"https://en.wikipedia.org/wiki/Centripetal_Catmull%E2%80%93Rom_spline\">\u003Cu>Catmull-Rom\u003C/u>\u003C/a> spline. This option was a bit simpler (where a knot is a simple position), but I figured that the runtime calculation would be fast enough. Once I replaced the data and the supporting code, the performance was much better and no \u003Ca href=\"https://docs.unity3d.com/Manual/performance-garbage-collector.html\">\u003Cu>GC\u003C/u>\u003C/a> was triggered.\u003C/p>\u003Cp>At this point, we had a spline that could be evaluated and from which line renderer positions could be generated. By doing this in an update loop (whenever a knot is moved) the line renderer is regenerated and the mesh updates accordingly.\u003C/p>",{"type":15,"items":111},[112],{"caption":113,"asset":114},"Each wire box represents a knot.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image14.png",{"type":12,"text":116},"\u003Cp>Once the line renderer was rigged with a spline, it was time to simulate the knots. For knots to behave like actual hair, some semi-proper physics had to be applied. When it comes to hair systems, spring physics is most often used. A quick Google search will yield thousands of examples, most of which use numerical integration methods to solve the systems (read up on the Euler and Verlet integration). The bottom line is that Verlet’s method provides better results but requires extra overhead, as multiple updates are needed per frame update.\u003C/p>\u003Cp>After some number-tweaking, I managed to create something like a heavy hair/gel hair system, so that the hair would behave statically. During initial testing I figured out that six to eight knots was enough. In the image above, the yellow ones are fixed (they can only rotate if needed/allowed), while the green ones can move. This was necessary in order for the strands to act like they were attached to the head. Without it, they would have moved and rotated freely. \u003C/p>\u003Cp>Spring logic handles distance and rotation constraints. In this case, the rotation constraint was set to 150 degrees, while the spring settings were stiff, in order to prevent stretching.\u003C/p>\u003Cp>However, once I actually started using multiple hair strands (approximately 30), the runtime calculations started causing hiccups. This was nothing major, but on lower devices it was still taking 10-20ms, which was enough to dip the fps (integration steps were in the 30s range). \u003Cbr>\u003Cbr>Thankfully, we live in an era of multiprocessors and parallel processing! We have an internal system for determining approximate device capability (based on CPU/GPU/mem) and even the devices we mark as “low” have at least a few CPU cores. Conveniently, Unity has a decent \u003Ca href=\"https://docs.unity3d.com/Manual/JobSystem.html\">\u003Cu>job system\u003C/u>\u003C/a>, which can utilize multiple CPUs. I ended up using \u003Ca href=\"https://docs.unity3d.com/ScriptReference/Unity.Jobs.IJobParallelFor.html\">\u003Cu>IJobParallelFor\u003C/u>\u003C/a>. \u003Cbr>\u003Cbr>In a nutshell, you provide a list of elements (via \u003Ca href=\"https://docs.unity3d.com/ScriptReference/Unity.Collections.NativeArray_1.html\">\u003Cu>NativeArray<>\u003C/u>\u003C/a>) and call up a schedule function. The internal system then does its magic and for each element in the list you get an Execute(index) call, giving you the index of the target element, on which you can now operate.\u003C/p>\u003Cp>Want to hear the cool part? Execute gets called from multiple threads. The concept is data isolation: each thread operates on a single array element, preventing data contention. Because the data is isolated, no locking mechanisms are needed. If you attempt to access data outside your assigned index, Unity will generate an error.\u003C/p>\u003Cp>The not-so-cool part? Nested containers can’t be used directly, meaning I cannot create a nice struct with a nested array of other structs and use that as a data element for the parallel job. It’s possible to manually create the structure handler/allocator, but that seemed like too much extra work to be worth the hassle. Another option I wanted to avoid was unsafe, which would give me full control and full responsibility over the memory. I would have to handle allocation, tracking and cleanup. Essentially, doing everything yourself (as in c/c++).\u003C/p>\u003Cp>If nested containers worked out of the box, you could write something like this:\u003C/p>",{"type":15,"items":118},[119],{"caption":22,"asset":120},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/screenshot-2025-06-06-at-15.40.18.png",{"type":12,"text":122},"\u003Cp>Thankfully, I found a reference to the \u003Ca href=\"https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeDisableContainerSafetyRestrictionAttribute.html\">\u003Cem>\u003Cu>NativeDisableContainerSafetyRestriction\u003C/u>\u003C/em>\u003Cu> \u003C/u>\u003C/a>attribute, which allowed me to access multiple data elements without warnings and errors being thrown up – not just my index ones. So, I ended up unrolling the data in an SoA (Structure Of Arrays) manner. I had one array for matrices (one per strand) one array for positions (strands \u003Cem> \u003C/em>max_positions\u003Cem>), \u003C/em>etc. If I needed to access positions for an nth strand, its starting position would be N\u003Cem> \u003C/em> max_positions. This was a bit messy but, once implemented, it wasn’t that bad. I just needed to be careful to stay within my index boundaries.\u003C/p>",{"type":15,"items":124},[125],{"caption":22,"asset":126},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/screenshot-2025-06-06-at-15.40.37.png",{"type":12,"text":128},"\u003Cp>Getting back to the job system, Schedule accepts a count (the number of data elements to be processed – total size) and preferred batch size. Batch size specifies how many elements will be processed at a time, on one thread. Basic logic would dictate that light work leads to a bigger batch size and heavy work means a smaller batch size. Since thread management and context switches all cost resources, running a job for single matrix multiplication doesn’t make sense. It also doesn’t make sense to run 100k multiplications in one job, since this might mean that the associated tasks aren’t evenly or optimally distributed (one job would run 100k, and the other only 2k). But it could make sense to run 256 of them. This was a tradeoff to balance the workload. Just to note, I ended up running a single hair strand on one job (so, batch size: 1). This gave me the overall best results.\u003C/p>\u003Cp>\u003Cbr>Once a job is scheduled, you need to wait for it to be completed. Only after it’s done can you safely resume data access. Between scheduling and waiting, thread workers are spun up, and “execute” is called for each data element. “Wait” is a blocking call, meaning your main thread will pause execution at this point, which is not ideal. Fortunately, you don't need to sit and wait at this point. Instead, you can schedule the job in Update and wait for it in LateUpdate. During this time, Unity runs some of its own workers (like animation). Once finished, you can submit the data for rendering. You just need to update LineRenderer data. All this is safe to do in LateUpdate, after the Wait. Accessing the data while the job system is running (between Run - Wait, Schedule - Complete) will trigger an exception.\u003Cbr>\u003C/p>\u003Cp>The downside to running in Update and waiting in \u003Ca href=\"https://docs.unity3d.com/ScriptReference/MonoBehaviour.LateUpdate.html\">\u003Cu>LateUpdate\u003C/u>\u003C/a> is that if you’re depending on animations (joints and their transforms), you’ll be stuck with an old transform state (local/world matrix), which can cause visual discrepancies (singleframe delays). In our case, this wasn’t an issue, since the animation didn’t cause any noticeable artifacts. But if we had any bigger problems, the only way to handle it would have been to run and wait for the jobs in LateUpdate.\u003C/p>",{"type":15,"items":130},[131,133],{"caption":22,"asset":132},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image20.png",{"caption":134,"asset":135},"Jobs running on a desktop machine, go Ryzen!","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image19.png",{"type":12,"text":137},"\u003Cp>With performance issues out of the way, it was finally time to move on!\u003C/p>\u003Cp>Next came the tools. We wanted to make sure to incorporate an array of different tools so that players could get creative. We went for:\u003C/p>\u003Cul>\u003Cli>\u003Cp>Comb/brush\u003C/p>\u003C/li>\u003Cli>\u003Cp>Scissors\u003C/p>\u003C/li>\u003Cli>\u003Cp>Magic growth potion\u003C/p>\u003C/li>\u003Cli>\u003Cp>Color sprays\u003C/p>\u003C/li>\u003Cli>\u003Cp>Accessories\u003C/p>\u003C/li>\u003C/ul>\u003Cp>The first (and most obvious) tool was a comb. Upon activating the tool, players could drag the tool with their fingers, moving around and over the hair strands, pulling them in the input direction. \u003C/p>\u003Cp>To accomplish this, I implemented a simple spherical pickup and release logic for knots. When a knot was in range, it was added to the manipulated list. When out of range, it was removed from the list. For each manipulated knot, I added a drag position offset based on user input, basically offsetting it in space. Initially, I implemented this by adding force to the knot, but this approach didn’t work well with quick swipes, since the time with which the knots were affected was too short. \u003C/p>\u003Cp>Later, we added a secondary comb (the precision one). The only big difference here was the pickup logic. It didn’t pick up all knots in range, but instead found the closest one and locked on to its parent hair strand, essentially allowing you to manipulate a single strand.\u003C/p>",{"type":15,"items":139},[140],{"caption":141,"asset":142},"Slightly visible red wire boxes, indicating which knots are being affected by the tool.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/lionhair100.gif",{"type":12,"text":144},"\u003Cp>Our second set of tools included a cutting tool (scissors) and growth spray (aka the “magical cure for baldness”). Of the two, the scissors were trickier to implement.\u003C/p>\u003Cp>First, I took the swipe action (start and end position of the user input) and got a 2D screen space line from it. Then I took a brute force approach. For each hair strand, I took the first and second vertex position and converted them into screen space. This gave me a line in the screen space. At that point, I’d have two lines on which I could do a line-to-line intersection. If an intersection was detected, I’d remove the dangling knots and push the last knot to the intersection position. If not, I’d take the second and the third vertex positions, and so on until the last vertex position had been reached. At that point, I’d proceed to the next strand.\u003C/p>",{"type":15,"items":146},[147],{"caption":22,"asset":148},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/lionhair101.gif",{"type":12,"text":150},"\u003Cp>The growth potion required the reverse process. Selected knots served as a list of affected hair strands. For each one, I started growing -> offsetting the last knot away from the previous one. Once a distance threshold was reached, I then created a new knot, and started moving that one away. This process was repeated until the maximum length of the strand was reached.\u003C/p>",{"type":15,"items":152},[153],{"caption":22,"asset":154},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/lionhair102.gif",{"type":12,"text":156},"\u003Cp>The last tool in the set is our color spray. To allow users to paint hair, I implemented a logic based on sphere collision (taking the color spray position from the user’s 2D space into the 3D world space) vs. line renderer position to determine which hair strand is affected, as well as which part of the strand. Then, I took that information and mirrored it on a pre-created color texture.\u003C/p>",{"type":15,"items":158},[159],{"caption":22,"asset":160},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/lionhair103.gif",{"type":12,"text":162},"\u003Cp>This was a hair-by-hair strand texture with a size of 64x16 RGB pixels. I wanted to keep the color textures as small as possible, while maintaining decent quality. Since strands are sort of rectangular in shape, I figured that the textures could also be that shape. Low resolution serves mainly two purposes: lower memory overhead (since we need ~30 textures) and lower CPU cost (due to the process of calculating which part of the texture is being painted over).\u003C/p>",{"type":15,"items":164},[165],{"caption":166,"asset":167},"Collection of runtime color textures.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image16.png",{"type":12,"text":169},"\u003Cp>The texture was then used in the line renderer material to multiply it with other textures. (We’ll look closer at the rendering setup below.) Due to the feature setup, once the hair was styled/cut/grown and the coloring stage had started, it’s not possible to go back and modify the hair again. This meant I could simplify the texture coloring, since I could just use the whole color texture independently of the strand length. At this point, color textures were only kept during feature activity (runtime). \u003C/p>\u003Cp>One nice feature that required almost no effort was the ability to use gradient colors. A color spray can define a number of colors (Unity’s Gradient) and, when coloring, the relative position offset on a strand can be used as the interpolator, which generates nice gradient patterns on the hair.\u003C/p>",{"type":15,"items":171},[172],{"caption":173,"asset":174},"A modern lion.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image18.png",{"type":12,"text":176},"\u003Cp>What about rendering? Well, one feature I found missing in LineRenderer was the ability to provide custom UVs with positions. This would have solved a few issues. Our final setup looked like this:\u003C/p>",{"type":15,"items":178},[179,182,185,188],{"caption":180,"asset":181},"Opacity texture ","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image10.png",{"caption":183,"asset":184},"Pattern texture","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image4.png",{"caption":186,"asset":187}," Color texture","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image28.png",{"caption":189,"asset":190},"Final result","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image1.png",{"type":12,"text":192},"\u003Cp>We have a base opacity texture, which we use to define the shape of the tip. Next, we have the pattern texture (monochrome) which defines the pattern on the strand. This pattern texture is then multiplied in shader with the color map, which is, by default, populated with a yellowish brown color. Once the user starts painting, that color is overridden. \u003Cbr>When hair got cut, we wanted the tip to stay the same shape. To achieve this, I simply moved the opacity texture UVs in the shader. The same logic applies when the hair is grown.\u003C/p>",{"type":15,"items":194},[195],{"caption":196,"asset":197},"Textures used in material.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image6.png",{"type":12,"text":199},"\u003Cp>In "normal" rendering, objects are rendered based on their order and depth using Z-depth testing. In the case of the hair, this would mean that strands pulled over the head would be rendered inside the head, making them partially invisible. To counteract this, we created two sets of strands, one back and one front. The back strands are rendered regularly, as we want them to be hidden behind the head, ears, etc. For the front ones we disabled regular Z-depth testing and implemented a higher render queue setting, forcing the strands to render over the head mesh. The one downside to this was that the strand “root” then also rendered over the head, meaning we’d have to position them really precisely, or hide them with something like fur.\u003C/p>",{"type":15,"items":201},[202,205],{"caption":203,"asset":204},"The split between front and back hair strands is clearly visible from the side.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image26.png",{"caption":206,"asset":207},"The front hair covers the ears, while the back hairs are rendered behind.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image27.png",{"type":12,"text":209},"\u003Cp>Now only one step remained: accessories. I programmatically determined (again, with simple radius checks) which knots I can enable the “decoration” slot (the circle icon) on. In addition to these, we’d agreed to have a few predetermined slots on the lion’s head.\u003C/p>",{"type":15,"items":211},[212],{"caption":213,"asset":214},"The “decoration” stage.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image23.png",{"type":12,"text":216},"\u003Cp>The circles show where the accessories can be placed. The user can pick them up and drag them to the desired location, where they lock into place. Tap/click can also be used to either add or remove accessories. \u003C/p>\u003Cp>After finishing up the main features and ironing out the majority of the bugs, our designers wanted one more thing. Even though this activity was fun and engaging, it still felt like something was missing. The actual access point for the hair salon activity is in the lion zone, on the island itself. There, users can see our mighty lion, and can even play with him. \u003C/p>",{"type":15,"items":218},[219],{"caption":22,"asset":220},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image17.png",{"type":12,"text":222},"\u003Cp>In the first instance, when you finished the hair salon activity, all you were left with was a memory of your incredibly artistic makeover. The lion on the island still had his old, boring mane. \u003C/p>\u003Cp>Why not simply transfer hair from the studio to the island, you ask? Well, it’s not quite that straightforward! The island lion’s mane was part of the lion mesh, which was a full 3D mesh rigged with a skeleton animator. Meanwhile, the lion in the activity was using “fake” hair based on 2D strands. So, there we had no simple way of transferring the look from one to the other.\u003C/p>\u003Cp>\u003Cbr>Still, I was asked if there was a way to at least transfer the painted colors to our island lion, if not the whole haircut/style. With the help of our 3D art team, we found a solution. I took the color textures from the hair strands the user painted in the activity and combined them into a single texture. To make this work, I had to order/mark all the strands in a sequential order, basically working clockwise, starting from the top.\u003C/p>",{"type":15,"items":224},[225,227],{"caption":226,"asset":174},"Colored hair strands.",{"caption":228,"asset":229},"The generated texture.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image9.png",{"type":12,"text":231},"\u003Cp>But then came the hard part! Usually, the first step would be to create the mesh, where each vertex can have one or more UV sets (coordinates), which then tell the rendering logic which part of the texture goes where on the mesh. Then the artists would paint these “blank” textures.\u003C/p>\u003Cp>In this case, though, we had to reverse the task. Since the texture was generated programmatically, our 3D team had to paint the UV set, so that it matched the texture, not the other way around, which isn’t a simple task. Still, our art team delivered – and now we have a colored lion.\u003C/p>",{"type":15,"items":233},[234],{"caption":235,"asset":236},"Color texture applied to the island lion mane.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/image13.png",{"type":12,"text":238},"\u003Cp>I feel a huge sense of accomplishment seeing this feature being used and enjoyed live. I’ve worked on many different game features over the years, but this one felt special because it covered so many different aspects – from design, tech and performance to rendering exploits and other magic. All in all, this was a great project with a great team. Why not download the game and give it a try?\u003C/p>","Marko Ravnjak, Senior Software Engineer",[],{"title":242,"copy":22,"backgroundImage":243,"cta":244},"The Tech Behind My Talking Hank's Hair Salon","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/mthi_hair_card_template.jpg",{"title":32,"type":33,"href":245},"/blog/tech/the-tech-behind-my-talking-hanks-hair-salon","the-tech-behind-my-talking-hanks-hair-salon",{"title":248,"description":7,"canonical":249,"robots":7},"The Tech Behind My Talking Hank's Hair Salon | Outfit7","http://localhost/blog/tech/the-tech-behind-my-talking-hanks-hair-salon",{"open_graph":40,"twitter":40,"site_name":41,"title":242,"description":42,"image":251},"https://cdn-o7.o7web.com/img/asset/YXNzZXRzLzRfYmxvZ19uZXdzL3RlY2hfYmxvZy9tdGhpX2hhaXJfc29jaWFsX2ltYWdlLmpwZw==?w=1200&h=630&q=70&fit=crop&s=a24aa357b667dd6157ed828b45c28ac5","content:tech:159f8d76-b27c-4cf1-8655-a8faa65f15cf.json","tech/159f8d76-b27c-4cf1-8655-a8faa65f15cf.json","tech/159f8d76-b27c-4cf1-8655-a8faa65f15cf",{"_path":256,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"date":257,"id":258,"collection":5,"content":259,"coverImage":299,"author":300,"categories":301,"card":306,"order":22,"slug":311,"title":307,"uri":310,"url":310,"meta":312,"canonical":22,"social":315,"_id":318,"_type":45,"_source":46,"_file":319,"_stem":320,"_extension":45},"/tech/08fe9043-91bb-436c-ba57-63217b5aaeab","2024-10-07T22:00:00.000000Z","08fe9043-91bb-436c-ba57-63217b5aaeab",[260,262,267,272,274,279,284,286,291,293,297],{"type":12,"text":261},"\u003Cp>\u003Cem>TL;DR: We’re taking a look at Outfit7’s art pipeline and how we transitioned from a proprietary system to the commercial solution — and a de facto industry standard — ShotGrid.\u003C/em>\u003C/p>\u003Cp>Hi! My name is Samy Ben Rabah and I’m a software engineer at Outfit7, working on the art pipeline between my coffee breaks. What’s an art pipeline? Read on to find out!\u003C/p>\u003Ch3>\u003Cstrong>What do you mean by “pipeline”?\u003C/strong>\u003C/h3>\u003Cp>Have you heard of the fun little cat who repeats absolutely everything you say and bugs you when he’s hungry? Did I hear “My Talking Tom”? Right you are! You’ve won two points!\u003C/p>\u003Cp>But how did this cheeky little feline manage to sneak into your smartphone? Was it through a backdoor or a cat flap? Nope. The answer is that Tom has literally been “pipelined” into your phone!\u003C/p>\u003Cp>At \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://outfit7.com/\">\u003Cu>Outfit7\u003C/u>\u003C/a>, we produce many games, including \u003Cstrong>My Talking Tom\u003C/strong> and \u003Cstrong>My Talking Angela\u003C/strong>. Each one is made up of many different elements, such as 3D characters, props, environments, animations, sounds, UI elements (backgrounds, buttons, and app icons), as well as promo material, ad banners and more to support the game’s promotion and distribution.\u003C/p>\u003Cp>Game developers implement the game logic and other mysterious bits and pieces, then everything gets packaged into an app that you install on your phone — et voilà!\u003C/p>\u003Cp>These elements are created by artists from different departments who intervene and contribute at different stages of the production. They use a collection of software (Autodesk Maya, Adobe Photoshop, SideFX Houdini, Adobe Substance Painter, etc.) to create assets, which are finally exported in a form that’s suitable for the game engine. Assets might be, for example, animations in Maya exported to .fbx, or multilanguage localisations exported to .png from a Photoshop file.\u003C/p>\u003Cp>To support these processes through the different stages of production, a management system is required, which we simply call the “pipeline.”\u003C/p>\u003Ch3>\u003Cstrong>Why do we need a “pipeline”?\u003C/strong>\u003C/h3>\u003Cp>Let’s look at a practical example to see why a “pipeline” is indispensable:\u003C/p>\u003Cp>Let’s say an animator is assigned an animation to work on.\u003C/p>",{"type":15,"items":263},[264],{"caption":265,"asset":266},"How do we get this Angela animation from Maya to the game engine?","https://cdn-o7.o7web.com/assets/angela-model.gif",{"type":15,"items":268},[269],{"caption":270,"asset":271},"Et voilà, our Angela animation is in the game engine!","https://cdn-o7.o7web.com/assets/angela-in-game.gif",{"type":12,"text":273},"\u003Cp>Typically, the animator would reference one or more external rigs (e.g. character, props) in Maya and animate them using diverse techniques (e.g. inverse kinematic, blend shapes, possibly some dynamics)\u003C/p>\u003Cp>Because the Maya animation is a working file, it cannot be directly used in the game. Therefore, it must be processed and exported in a form that’s suitable for the game engine:\u003C/p>\u003Cul>\u003Cli>\u003Cp>Rig joints need to be baked.\u003C/p>\u003C/li>\u003Cli>\u003Cp>The rig needs to be stripped of unnecessary nodes, some nodes might be renamed, etc.\u003C/p>\u003C/li>\u003Cli>\u003Cp>The baked animation is then exported to .fbx in a specific location with a specific naming convention. If we have multiple rigs, they’re each exported separately using specific naming conventions.\u003C/p>\u003C/li>\u003Cli>\u003Cp>And so on…\u003C/p>\u003C/li>\u003C/ul>\u003Cp>Obviously, these operations would be done by an exporter script that the animator would run and voilà, we have our animation in the game engine! Beautifully simple, isn’t it?\u003C/p>\u003Cp>But, before we can reach this export stage, we need to take some aspirin and ask ourselves a few simple questions:\u003C/p>\u003Cp>1. How do we organise and keep track of our animations (and all the other assets)? Do we use a big colourful spreadsheet to organize these tasks by category/task name?\u003C/p>\u003Cp>2. How does the animator “publish” a new version? Does she work locally and then copy the file in a predetermined network location using an agreed naming convention?\u003C/p>\u003Cp>3. If, by accident, two animators work on the same file, how would the conflict be resolved — or prevented in the first place?\u003C/p>\u003Cp>4. The animation might use one or more rig Maya files. How does the animator know where to find these? How does the animator update the rigs? Does she browse the file system and search for updates? How does she know when updates are available?\u003C/p>\u003Cp>5. All the referenced files (rigs, textures, etc.) would normally be copied locally instead of being directly referenced from the network. How does the artist know what files this Maya scene depends on?\u003C/p>\u003Cp>6. If we would like to have these files on the cloud, how would artists fetch them and how would they upload their published files?\u003C/p>\u003Cp>7. How does the animator know where export scripts are located? Would she copy them locally? How would she get updates?\u003C/p>\u003Cp>8. If the animation exporter has settings specific to a project, how can we make sure the animator uses the proper settings? Do we provide a different exporter for each project with hardcoded settings? Or are they set manually?\u003C/p>\u003Cp>9. How can we track what the animator is working on? How can we schedule the tasks she’s working on? Should we use a spreadsheet?\u003C/p>\u003Cp>And on and on \u003Cem>ad infinitum. \u003C/em>Do you have a headache yet? If not, you should!\u003C/p>\u003Cp>Without a system taking charge of all these processes, and relying on people manually taking care of all the details above (and many more), it’s safe to say we’re doomed.\u003C/p>\u003Cp>And this is where our pipeline comes into play…\u003C/p>\u003Cp>Here’s what the pipeline can do for us:\u003C/p>\u003Col>\u003Cli>\u003Cp>\u003Cstrong>Project management, organization of assets: \u003C/strong>The pipeline helps to organize projects and assets in a logical structure. Systematic organization allows for predictable asset location and naming conventions\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>Asset centralization: \u003C/strong>Assets are published, versioned and stored on the network or remote storage, ensuring they’re backed up and accessible when needed.\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>Metadata publishing\u003C/strong>:The pipeline stores metadata associated with published content, such as the creator, description, file size, etc.\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>Dependencies:\u003C/strong> The pipeline keeps track of dependencies. In the example above, a published Maya animation file would have information about the rigs it references and each published rig would have links to textures it references, and so on. If you were to pull this Maya file up on your machine, it would be possible to recursively go through all the dependencies and collate all the files required to get a functional scene.\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>Integrations, toolsets, unified workflows\u003C/strong>: Typically, a pipeline will have integrations in various applications, such as Photoshop, Houdini and/or Maya and will have an API/toolkit allowing you to customise or write new integrations. A dedicated UI would allow the user to navigate through the assets without having to know where the actual files reside on the network or cloud. This also makes it easier to get a unified set of tools, such as exporters. This allows for the unification of workflows, resulting in reproducible outputs.\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>User management\u003C/strong>: The pipeline helps to manage users, and assign them to projects and tasks. It also allows you to assign permissions with various levels of granularity. For example, you can restrict users to specific projects.\u003C/p>\u003C/li>\u003Cli>\u003Cp>\u003Cstrong>Project management, planning, tracking\u003C/strong>:\u003Cbr>Artists are assigned tasks, which can be scheduled in a global plan. For instance, before it can be animated, a character needs to go through a concept/model/rigging phase. Such tasks can be planned chronologically in a Gantt chart.\u003C/p>\u003C/li>\u003C/ol>\u003Cp>Without a pipeline, trivial operations, such as naming files and saving them in the correct location, would have to be done manually, inevitably leading to errors and making the whole production extremely painful.\u003C/p>\u003Ch3>\u003Cstrong>Outfit7 art pipeline\u003C/strong>\u003C/h3>\u003Cp>The Outfit7 art pipeline is currently undergoing a transition from a proprietary system to ShotGrid, a commercial solution and de facto industry standard.\u003C/p>\u003Ch3>\u003Cstrong>Juggernaut: the old way\u003C/strong>\u003C/h3>\u003Cp>Our in-house pipeline, which we call “Juggernaut” (actually the frontend UI) was developed over several years. We could say that it was developed “organically” with some planning and design here and there. But in reality it was much more like a hot potato…\u003C/p>\u003Cp>Juggernaut revolves around the following components:\u003C/p>\u003Cul>\u003Cli>\u003Cp>a backend powered by Django and a MySQL database\u003C/p>\u003C/li>\u003Cli>\u003Cp>a file server mounted on artists’ machines\u003C/p>\u003C/li>\u003Cli>\u003Cp>a Python/PySide2 frontend (Juggernaut) used by artists to publish their work\u003C/p>\u003C/li>\u003Cli>\u003Cp>a collection of exporters (Photoshop, Maya, etc.)\u003C/p>\u003C/li>\u003Cli>\u003Cp>a web UI for project creation and user assignment\u003C/p>\u003C/li>\u003C/ul>",{"type":15,"items":275},[276],{"caption":277,"asset":278},"Juggernaut pipeline","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/art-pipeline-juggernaut-pipeline.jpg",{"type":15,"items":280},[281],{"caption":282,"asset":283},"Juggernaut","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/art-pipeline-juggernaut.jpg",{"type":12,"text":285},"\u003Cp>In a nutshell, Juggernaut works as follows: Each project is organised into Project/Modules/Categories/Tasks/FileVersions. Artists work on tasks (UI elements in Photoshop, 3D models, rigs, Maya animations, Substance Painter textures, etc.). These artists work locally and publish versioned files, which are copied to the server. Then, they export various files using custom exporters:\u003C/p>\u003Cul>\u003Cli>\u003Cp>game exports (to Unity or Starlite, our proprietary game engine) such as app icons, geometry/rigs/animations to .fbx, etc.\u003C/p>\u003C/li>\u003Cli>\u003Cp>promo materials for campaigns, ad banners, thumbnails, etc.\u003C/p>\u003C/li>\u003C/ul>\u003Ch3>\u003Cstrong>File system\u003C/strong>\u003C/h3>\u003Cp>In our in-house pipeline, everyone had write access to the file server, resulting in a lot of “dark matter” accumulating over the years: gigabytes of data whose mysterious presence could be felt, without anyone being able to tell what it was used for. To make things worse, there were also cases of users editing published files directly or accidentally overwriting them — or even worse, corrupting the file and making it unreadable.\u003C/p>\u003Ch3>\u003Cstrong>The external pipeline\u003C/strong>\u003C/h3>\u003Cp>Initially, Juggernaut was designed to work only from within our intranet. This required a VPN connection in order to access it from outside. It also required a lot of maintenance, and also posed a potential security risk when giving access to external artists. Moreover, the performance was really poor because files were accessed over Samba.\u003C/p>\u003Cp>A system called the “external pipeline” was then developed to allow external users to use Juggernaut via an external site with database replication and using a local server. The development of this system was accelerated when everyone started to work remotely during the Covid-19 pandemic.\u003C/p>\u003Cp>Over the years, Juggernaut has served its purpose. It’s been used for many games, despite its limitations and bugs. A lot of our time was spent fixing bugs and supporting users instead of focusing on new features.\u003C/p>\u003Cp>Additionally, this pipeline only tracked published file information, but it lacked tools to support project management, task tracking and reviewing.\u003C/p>\u003Cp>So, in mid-2021 we decided to abandon Juggernaut and adopt ShotGrid for our pipeline.\u003C/p>\u003Ch3>\u003Cstrong>ShotGrid: the new way\u003C/strong>\u003C/h3>",{"type":15,"items":287},[288],{"caption":289,"asset":290},"ShotGrid Desktop","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/art-pipeline-shotgrid-desktop.jpg",{"type":12,"text":292},"\u003Cp>Autodesk \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://www.shotgridsoftware.com/\">\u003Cu>ShotGrid\u003C/u>\u003C/a> (formerly known as Shotgun) is a production management system with integrations with most standard digital content creation (DCC) software and extensive APIs. Its main features and advantages are:\u003C/p>\u003Cul>\u003Cli>\u003Cp>production tracking, task planning, Gantt, reviewing\u003C/p>\u003C/li>\u003Cli>\u003Cp>user management, high granularity permissions system\u003C/p>\u003C/li>\u003Cli>\u003Cp>it’s highly customisable, thanks to its \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://developer.shotgridsoftware.com/\">\u003Cu>Python/REST APIS and Shotgrid Toolkit\u003C/u>\u003C/a> and customisable schema\u003C/p>\u003C/li>\u003Cli>\u003Cp>customisable web pages and views\u003C/p>\u003C/li>\u003Cli>\u003Cp>advanced search and filtering\u003C/p>\u003C/li>\u003Cli>\u003Cp>extensive developer documentation/samples/git repos/community/forums\u003C/p>\u003C/li>\u003Cli>\u003Cp>industry standard: it makes it easy to enrol new users\u003C/p>\u003C/li>\u003Cli>\u003Cp>good customer support\u003C/p>\u003C/li>\u003C/ul>\u003Cp>We began development on Shotgrid in mid-2021, and the transition required the following developments:\u003C/p>\u003Cp>Pipeline development:\u003C/p>\u003Cul>\u003Cli>\u003Cp>adapting the base pipeline configuration as well project structure to meet our needs\u003C/p>\u003C/li>\u003Cli>\u003Cp>writing new frameworks/apps and custom hooks (see “hooks” section)\u003C/p>\u003C/li>\u003Cli>\u003Cp>refactoring exporters\u003C/p>\u003C/li>\u003Cli>\u003Cp>Unity integration to access ShotGrid\u003C/p>\u003C/li>\u003Cli>\u003Cp>developing a client UI to download published files from AWS S3 bucket\u003C/p>\u003C/li>\u003Cli>\u003Cp>writing Event Daemon scripts (triggered upon ShotGrid events)\u003C/p>\u003C/li>\u003Cli>\u003Cp>migrating our existing projects to ShotGrid\u003C/p>\u003C/li>\u003C/ul>\u003Cp>Production:\u003C/p>\u003Cul>\u003Cli>\u003Cp>leveraging ShotGrid’s powerful production management capabilities such as task assignment, tracking, review system, custom pages.\u003C/p>\u003C/li>\u003Cli>\u003Cp>onboarding users to ShotGrid and changing workflows\u003C/p>\u003C/li>\u003Cli>\u003Cp>migrate previous tools (Trello, spreadsheets) to ShotGrid pages\u003C/p>\u003C/li>\u003C/ul>\u003Ch2>\u003Cstrong>Pipeline development\u003C/strong>\u003C/h2>\u003Ch3>\u003Cstrong>Custom entities\u003C/strong>\u003C/h3>\u003Cp>ShotGrid is quite flexible and lets you update the database schema by creating \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://help.autodesk.com/view/SGSUB/ENU/?guid=SG_Producer_pr_project_tracking_pr_entities_html\">\u003Cu>custom entities\u003C/u>\u003C/a>, as well as adding and editing fields of different types (text, float, int, boolean, entity fields etc.) We used these to create new entities, such as Categories or ApplicationSettings (holding config JSON data) and also added fields to existing entities.\u003C/p>\u003Ch3>\u003Cstrong>ShotGrid Configuration\u003C/strong>\u003C/h3>\u003Cp>We started with the \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://github.com/shotgunsoftware/tk-config-default2\">\u003Cu>default toolkit pipeline configuration\u003C/u>\u003C/a> and adapted it to fit our needs using the ShotGrid APIs: The ShotGrid Python API and rest API was used to interact with the ShotGrid database to create/update/delete/edit entities. The Shotgrid toolkit API was used for DCC app integrations.\u003C/p>\u003Cp>We learned all about the inner workings of ShotGrid (and some of its limitations) while developing new features.\u003C/p>\u003Ch3>\u003Cstrong>Files on the cloud\u003C/strong>\u003C/h3>\u003Cp>Early on, we determined that we wanted to have our files on the cloud. Although the ShotGrid toolkit doesn’t have a built-in mechanism to deal with cloud-based files, it does allow you to write frameworks and hooks to achieve it. We developed a remote storage framework to handle file transfer to our Amazon S3 bucket for exactly this purpose.\u003C/p>\u003Ch3>\u003Cstrong>Custom hooks\u003C/strong>\u003C/h3>\u003Cp>The ShotGrid toolkit has a nice feature where the apps, engine or framework can be extended via hooks, which are Python functions that override and extend the base class behaviour.\u003C/p>\u003Cp>Each app has a YAML configuration file, where you can specify an inheritance chain for a specific engine and context, and define where the Python code will reside.\u003C/p>\u003Cp>For instance, the publish collector (which is the phase where the files that need to be published are collected) was extended to meet our specific needs, and are engine-specific.\u003C/p>\u003Cpre>\u003Ccode>settings.tk-multi-publish2.maya.asset_step:\ncollector: "{self}/collector.py:{engine}/tk-multi-publish2/basic/collector.py:{config}/tk-maya/tk-multi-publish2/basic/collector.py"\n# the {config}/tk-maya/tk-multi-publish2/basic/collector.py part adds Maya \n# specific code for the maya publish collector\u003C/code>\u003C/pre>\u003Cp>For example, you can:\u003C/p>\u003Cul>\u003Cli>\u003Cp>collect linked smart objects in Photoshop files\u003C/p>\u003C/li>\u003Cli>\u003Cp>collect textures, rig exports, 3D geometry\u003C/p>\u003C/li>\u003Cli>\u003Cp>collect animation .fbx exports and playblasts\u003C/p>\u003C/li>\u003C/ul>\u003Ch3>\u003Cstrong>Pre-publish validation\u003C/strong>\u003C/h3>\u003Cp>Another example of a hook is the validation stage. This is an opportunity to run some checks before allowing the publishing to go through, preventing problems down the line and resulting in cleaner published data. This was a major improvement in comparison to our in-house pipeline.\u003C/p>\u003Cp>For example, you can:\u003C/p>\u003Cul>\u003Cli>\u003Cp>check if the user added a comment/thumbnail to the published file\u003C/p>\u003C/li>\u003Cli>\u003Cp>check if there’s enough disk space to accommodate the new publishes and exports\u003C/p>\u003C/li>\u003Cli>\u003Cp>check if a referenced Maya file/texture/linked smart object is under the project root\u003C/p>\u003C/li>\u003Cli>\u003Cp>check if a rig conforms to the expected structure and naming convention\u003C/p>\u003C/li>\u003Cli>\u003Cp>plus many more possibilities!\u003C/p>\u003C/li>\u003C/ul>\u003Ch3>\u003Cstrong>Publish finalize\u003C/strong>\u003C/h3>\u003Cp>Another example of hook is the publish finalize, which is invoked after the publish is completed. Since our files are stored on Amazon, this is where the actual upload of the published files starts.\u003C/p>\u003Ch3>\u003Cstrong>Event Daemons\u003C/strong>\u003C/h3>\u003Cp>ShotGrid has an event stream where nearly every operation on the ShotGrid database is recorded. It’s therefore possible via an Event Daemon or a web hook to execute some operations upon certain events. We deployed an Event Daemon (running on some container somewhere in cyberspace) and are adding scripts, such as:\u003C/p>\u003Cul>\u003Cli>\u003Cp>sending an alert when a duplicate category or asset is created\u003C/p>\u003C/li>\u003Cli>\u003Cp>sending a Slack notification when someone leaves a note on your version\u003C/p>\u003C/li>\u003Cli>\u003Cp>and more\u003C/p>\u003C/li>\u003C/ul>\u003Ch3>\u003Cstrong>Apps\u003C/strong>\u003C/h3>\u003Cp>The ShotGrid toolkit allows you to create new apps that can be started from the ShotGrid Desktop and which communicate with ShotGrid. For example, a disk space management app was created for artists to safely clean up their published files (especially problematic Photoshop files under 1.5GB).\u003C/p>\u003Cp>Another app (currently in the making) is a Maya batch exporter allowing you to send Maya batch exports on a remote machine from a published Maya scene.\u003C/p>\u003Ch3>\u003Cstrong>Frameworks\u003C/strong>\u003C/h3>\u003Cp>As mentioned earlier, we wrote custom frameworks, which are libraries that can be reused in apps and engines.\u003C/p>\u003Cp>Among others, we created:\u003C/p>\u003Cul>\u003Cli>\u003Cp>a remote storage framework to handle the file transfers to an Amazon S3 bucket\u003C/p>\u003C/li>\u003Cli>\u003Cp>an ExtendScript-library framework containing our ExtendScript libraries and exporters for Photoshop, as well as binaries such as ImageMagick\u003C/p>\u003C/li>\u003Cli>\u003Cp>a common library framework with common code such as ShotGrid API utilities.\u003C/p>\u003C/li>\u003Cli>\u003Cp>a Maya framework with Maya-specific functions as well as shelves, plug-ins, etc.\u003C/p>\u003C/li>\u003C/ul>\u003Cp>Maya plug-ins, scripts and shelves are bundled in the Maya framework and all the setup (such as Maya script/plugin/module paths environment variables, directory remapping, default units, etc) is done “under the hood” when users launch Maya from the ShotGrid desktop. Without such a system, artists would have to set up everything manually, which would involve needing pipeline developers to assist them at some point. It would also result in discrepancies in artists’ setups and it would also be difficult to push updates.\u003C/p>\u003Ch3>\u003Cstrong>Deployment\u003C/strong>\u003C/h3>\u003Cp>The ShotGrid toolkit defines the project setup, such as file system locations and templates, as well as frameworks/apps/engines and their versions in a \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://developer.shotgridsoftware.com/37f575b8/\">\u003Cu>pipeline configuration\u003C/u>\u003C/a>. We deploy our frameworks in beta configurations that artists can test before updating the master configuration. This is really quite an improvement compared to what we used to have with our in-house pipeline and Juggernaut.\u003C/p>",{"type":15,"items":294},[295],{"caption":22,"asset":296},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/art-pipeline-ibi-4.jpg",{"type":12,"text":298},"\u003Cp>As far as the deployment is concerned, we made our lives easier by writing config management tools to easily upload updated frameworks and generate new configurations and propagate them to all the projects (we currently have nearly 100 projects at the time of writing…)\u003C/p>\u003Ch3>\u003Cstrong>Old project migration\u003C/strong>\u003C/h3>\u003Cp>We migrated a lot of old projects from our old pipeline to ShotGrid and, given the scope of the task, it had to be automated. For this purpose, we developed a migration tool using the ShotGrid Python API.\u003C/p>\u003Cp>The process consisted of the following steps:\u003C/p>\u003Cul>\u003Cli>\u003Cp>scanning the Juggernaut project to generate a big data structure containing all the publish metadata: categories/tasks/file versions, application settings\u003C/p>\u003C/li>\u003Cli>\u003Cp>mapping the old file versions to their ShotGrid PublishedFile equivalents and populating the destination ShotGrid project (categories, assets, tasks, PublishedFiles, versions, thumbnails, etc.)\u003C/p>\u003C/li>\u003Cli>\u003Cp>recording PublishedFile upstream dependencies\u003C/p>\u003C/li>\u003Cli>\u003Cp>migrating application settings (stored as JSON data) to their ShotGrid equivalents (using \u003Ca rel=\"noopener ugc nofollow\" target=\"_blank\" href=\"https://knowledge.autodesk.com/support/shotgrid/learn-explore/caas/CloudHelp/cloudhelp/ENU/SG-Administrator/files/ar-get-started/SG-Administrator-ar-get-started-ar-enabling-custom-entities-html-html.html\">\u003Cu>custom entities\u003C/u>\u003C/a>).\u003C/p>\u003C/li>\u003Cli>\u003Cp>uploading the files to our AWS S3 bucket.\u003C/p>\u003C/li>\u003C/ul>\u003Ch3>\u003Cstrong>Maya files and references\u003C/strong>\u003C/h3>\u003Cp>The biggest challenge was dealing with Maya files containing references to external files (such as other Maya files, textures or audio files), which then had to be remapped to match the ShotGrid paths.\u003C/p>\u003Cp>Given the number of files, it was unrealistic (if not impossible) to run Maya batch processes to substitute these paths. Fortunately, all our Maya files were in the ASCII format, and a simple text substitution was done (thank you, regular expressions!), which worked well.\u003C/p>\u003Cp>Additionally, upstream dependencies were also recorded. These dependencies are, for example, other Maya scenes or textures that a Maya scene depends on. This information is crucial in order to download a fully functional Maya scene.\u003C/p>\u003Cp>The final step was to upload the new published files to an AWS S3 bucket.\u003C/p>\u003Cp>We had around 25 projects to migrate and each was spun off into a game and various distribution projects. This resulted in a total of around 50 ShotGrid projects. An initial pass was done to migrate the bulk of the projects (while we were still using the old pipeline) and, later on, incremental migrations were done for the final switch.\u003C/p>\u003Cp>Here are some numbers to illustrate the scope of the migration:\u003C/p>\u003Cul>\u003Cli>\u003Cp>~ 50 projects\u003C/p>\u003C/li>\u003Cli>\u003Cp>~ 29k assets\u003C/p>\u003C/li>\u003Cli>\u003Cp>~ 290k migrated files\u003C/p>\u003C/li>\u003Cli>\u003Cp>~ 14TB of data\u003C/p>\u003C/li>\u003C/ul>\u003Cp>Some artists tested and validated the migrated data on samples and also carried out test exports to confirm the migration was backward compatible and producing the expected outputs.\u003C/p>\u003Ch3>\u003Cstrong>Conclusion\u003C/strong>\u003C/h3>\u003Cp>According to experts, the wheel was invented a few thousand years ago. So why reinvent it? Why not use an established and proven system and focus on the important bits instead?\u003C/p>\u003Cp>Although our old pipeline helped us produce many games, it lacked flexibility, had limitations and it used a good chunk of our resources to duct tape over issues and fix bugs instead of spending our time developing new features.\u003C/p>\u003Cp>In this context, the switch to ShotGrid made sense. Thanks to its solid foundations and flexibility, it’s definitely a plus for our pipeline and for the production of our future games.\u003C/p>\u003Cp>The switch to ShotGrid took longer than expected, but the advantage is that we had time to consolidate the codebase. Moreover, our artists are now more familiar with ShotGrid and the new workflows. They’re progressively getting rid of their ingrained habits, built over years on our old pipeline.\u003C/p>\u003Cp>We’re now in the process of shutting down our old pipeline and fully switching over to ShotGrid. So, RIP Juggernaut. Welcome, ShotGrid!\u003C/p>","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/art-pipeline-cover.jpg","Samy Ben Rabah, Senior Software Engineer",[302],{"id":303,"title":304,"slug":305},"tech_categories::game-dev","Game Dev","game-dev",{"title":307,"copy":22,"backgroundImage":308,"cta":309},"Outfit7’s Art Pipeline","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/web_card_09.10.2024.jpg",{"title":32,"type":33,"href":310},"/blog/tech/outfit7s-art-pipeline","outfit7s-art-pipeline",{"title":313,"description":7,"canonical":314,"robots":7},"Outfit7’s Art Pipeline | Outfit7","http://localhost/blog/tech/outfit7s-art-pipeline",{"open_graph":40,"twitter":40,"site_name":41,"title":307,"description":316,"image":317},"How we transitioned to ShotGrid","https://cdn-o7.o7web.com/img/asset/YXNzZXRzLzRfYmxvZ19uZXdzL3RlY2hfYmxvZy9zb2NpYWxfaW1hZ2VfMDkuMTAuMjAyNC5qcGc=?w=1200&h=630&q=70&fit=crop&s=cf01468b631e3b835faa859b57e00f48","content:tech:08fe9043-91bb-436c-ba57-63217b5aaeab.json","tech/08fe9043-91bb-436c-ba57-63217b5aaeab.json","tech/08fe9043-91bb-436c-ba57-63217b5aaeab",{"_path":322,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"date":323,"id":324,"collection":5,"content":325,"coverImage":7,"author":376,"categories":377,"card":382,"order":22,"slug":387,"title":383,"uri":386,"url":386,"meta":388,"canonical":22,"social":391,"_id":394,"_type":45,"_source":46,"_file":395,"_stem":396,"_extension":45},"/tech/c8e19920-6583-42f3-bcff-a71e063f4030","2024-09-25T22:00:00.000000Z","c8e19920-6583-42f3-bcff-a71e063f4030",[326,328,333,335,340,342,346,348,353,355,360,362,367,369,374],{"type":12,"text":327},"\u003Cp>\u003Cem>This is the second instalment of our deep dive into the world of AB testing. To read more, you can find Part 1 \u003Ca href=\"https://outfit7.com/blog/tech/lessons-learned-from-10-years-of-a-b-testing-part-1\">here\u003C/a>.\u003C/em>\u003C/p>\u003Cp>By following proven testing processes, we can avoid wasting time on A/B tests that don’t serve up actionable results. In this section, I’ll share the process that works well for us. \u003C/p>\u003Cp>The first step in the process is to identify the problem that we want to solve, along with an associated metric that we would like to improve. Next, we generate ideas that address the problem, selecting the most promising one. Then, we implement it as an A/B test and start collecting data. When enough data is collected, we analyze it and prepare our test report. Finally, decisions are made based on the results of the A/B test. If the test confirms that the new version of the product outperforms the current one, we switch to the new one. Otherwise, we go back to one of the previous steps.\u003C/p>",{"type":15,"items":329},[330],{"caption":331,"asset":332},"Our A/B test process steps.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-8.png",{"type":12,"text":334},"\u003Ch3>Identifying problems\u003C/h3>\u003Cp>Starting with a clear problem increases the chances of ending up with useful results that will help us make good decisions. \u003C/p>\u003Cp>The problem might be related to business (e.g. the number of users is too low to cover the development costs), product (e.g. user interface or product features are missing or poorly designed), or monetisation (e.g. users do not make repeated purchases). \u003C/p>\u003Cp>Problems can be identified by various people in the team. In our case, they’re most often flagged by the product team that designs games and new game features, or by the analytics team that regularly checks the performance of games and features. Business problems are often detected by the controlling team, technical problems by the development and quality assurance teams, and some other problems are reported by users themselves.\u003C/p>\u003Cp>A problem must be translated into a metric that will indicate whether the new version of the product fixed the problem or not. We can measure things like revenue by sources, the number of daily or weekly active users, lifetime value of a user, crash rate, user retention, daily time spent playing the game, share of users that make the first and repeated in-app purchases, etc. Oftentimes, more than one metric is observed. For instance, a goal might be to decrease the prices while increasing the number of paying users, so that the total revenue increases or remains stable. Some things cannot be measured directly, in which case a proxy measure needs to be used instead.\u003C/p>\u003Cp>As you begin conducting regular A/B tests, people will naturally start suggesting possible new tests. Always remember to take a step back before rushing into testing. Consider which problem the proposed test is trying to solve. Establish whether this is the real problem or whether there  is actually a bigger issue or underlying problem that needs to be solved first. From there, you can start considering the best approach to fixing the problem. By taking the time to consider the issue from all angles, you may come up with an alternative solution, or one that will solve a bigger problem, be cheaper to execute, or provide more useful data.\u003C/p>\u003Ch3>Generating and evaluating test ideas\u003C/h3>\u003Cp>How you generate A/B test ideas depends on the problem you are trying to solve. Often, it’s beneficial to brainstorm with an interdisciplinary team in order to consider different aspects of the problem and potential solutions and to generate diverse ideas. Rough ideas can then be refined or adjusted depending on goals and available resources (time, budget, acceptable risk level).\u003C/p>\u003Cp>In order to decide which A/B test should be done, it’s important to consider several factors: How big is the expected impact on top business objectives? Will the difference between the test and control groups be measurable with the available number of users? What will the total cost of the test be? Is there enough time to execute the test and make a final decision? \u003C/p>\u003Ch3>Impact\u003C/h3>\u003Cp>Estimating the expected impact of your A/B test will allow you to compare the value of investing in the test versus investing in other opportunities. At Outfit7, we use a key performance indicators (KPI) tree to get reliable estimates. KPI trees visually illustrate the relationship between high-level and lower-level KPIs. For example, we can compute the daily revenue of a game (the root of our KPI tree) by multiplying the number of daily active users (left branch) with the average daily revenue per user (right branch). Each KPI can be further broken down into lower-level KPIs as needed. It can be challenging to estimate the impact of a product change on total revenue, for example, so we instead estimate the impact on lower-level KPIs and cascade those estimations up the KPI tree. \u003C/p>\u003Cp>Consider an A/B test where we decide not to display an in-game banner ad if the advertiser's offer falls below a minimum acceptable price threshold. A monetization expert predicts that this action could raise the average banner ad price by 5%. However, the overall revenue increase would be less. By rejecting low-price banner ads, the number of ads shown daily would decrease, resulting in a projected increase of only 2.9% in daily banner ad revenue per user. It's worth noting that banner ads are just one revenue stream, alongside interstitials, video ads, and in-app purchases, which remain unaffected by the test. Consequently, the daily revenue per user would only rise by 0.8%. Conversely, showing fewer ads might boost user retention and thus the daily active user count. Taking these factors into account, the anticipated total revenue increase from the test is estimated to be around 1.8%.KPI tree example: rejecting cheap banner ads affects several KPIs.\u003C/p>",{"type":15,"items":336},[337],{"caption":338,"asset":339},"KPI tree example: rejecting cheap banner ads affects several KPIs.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-9.png",{"type":12,"text":341},"\u003Cp>But is it worth investing in an A/B test with such potential? It does not depend only on the relative difference but also on the absolute values. For example, if the total daily revenue is $1000, we expect an additional $18 profit per day, or $6,400 per year. The additional revenue would cover the cost of roughly one employee for one month of work, so it does not make sense to do the test if implementing, running, and analyzing it will require more than one month of work. On the other hand, if daily revenue is $100,000, we are potentially looking at an additional $1,800 per day, which will easily cover the cost of the test in just a short period of time.\u003C/p>\u003Ch3>Test power\u003C/h3>\u003Cp>If an A/B test idea has potential for Outfit7, we proceed first by estimating whether we’ll actually be able to measure the difference between the test groups. We do this by calculating the power of the statistical test that will be used to compare variants. The power depends on three things: how big the impact of variant B is on the observed metric, how big the differences between individual users are, and how many test users we will have. \u003C/p>\u003Cp>Let’s assume that the goal of a test is to improve daily play time. It would be easy to detect the difference between test groups if all users had daily play time in a narrow range (e.g. 20 and 23 minutes), group B increased their average daily play time by a significant amount (e.g. from 22 to 32 minutes) and the number of test users was high (e.g. 1M per test group). On the other hand, it would be difficult to detect the difference if daily play times were evenly distributed over a wide range (e.g. 2 to 60 minutes),  the increase in group B was small (e.g. from 23 to 23.1 minutes) and we only had a small group of test users (e.g. 30 per group). \u003C/p>\u003Cp>Take a look at the overlaps in the distribution of measurements in the two test groups shown below. With a big overlap, it becomes hard to detect the difference, (i.e. you will need a lot of test users to get a statistically significant result).\u003C/p>",{"type":15,"items":343},[344],{"caption":22,"asset":345},"https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-10.png",{"type":12,"text":347},"\u003Cp>Still, we only have some control over two out of the three parameters. We can change the difference between the two groups by making the tested variant more extreme (e.g. increasing the amount of rewards by 200% instead of 10%). A/B tests like this are useful because they measure the maximal impact of a parameter on the performance of the product. Tests with more realistic variants include a test group with acceptable change and an alternative group with a bit bigger (but not extreme) change, or an equally big change in the opposite direction.\u003C/p>\u003Cp>We can also control the number of test users. If the expected difference between test groups will be hard to detect, we must increase the number of test users. However, the required number of test users often gets far beyond the number of users that can actually be recruited for the test.\u003C/p>\u003Cp>Therefore, you should always use a test power calculator to estimate the number of required test users before you start implementing an A/B test. It’s not worth running  underpowered tests (tests with too few test users) because you will end up with inconclusive results.\u003C/p>\u003Ch3>Cost\u003C/h3>\u003Cp>The third check we do before running an A/B test is cost estimation. On top of the obvious costs of developing a new version of the product, data collection, cleaning and analysis, and test-user acquisition there are also hidden costs. \u003C/p>\u003Cp>The first one is the opportunity cost. Regardless of your test results, some of the participants will inevitably be using an inferior version of your product during the test, resulting in worse user experience and decreased monetization.\u003C/p>\u003Cp>The second hidden cost is the potential negative impact on test users, the entire user community, or even your brand as a whole. For example, our increased app size test described above had a significant negative impact on the game revenue. We decreased the number of new players and lost a lot of existing players during that test. We were aware of the risk before the test, therefore we limited it to an appropriate market segment and monitored its impact in order to stop the test as soon as the results were clear.\u003C/p>\u003Cp>There may also be some hidden costs associated with cleaning up after a test. Namely, what do you do with the users from the test groups that did not win? If it’s easy to switch them to the winning group, there might be almost no cost. In mobile gaming, for example, it’s often just a matter of changing a configuration parameter and pushing it to client devices. However, sometimes it’s hard – or even impossible – to switch the users to the winning group. In this case, you have to bear the cost of some users having an inferior version of the product (that you may need to keep supporting) until the end of its lifetime.\u003C/p>\u003Ch3>Time needed\u003C/h3>\u003Cp>The final consideration is whether you can have the results of the test before the related decision needs to be taken. Even with all our A/B test infrastructure and experience, I’ve never seen a test go from concept to outcome in under two weeks, and I saw plenty that took over half a year.\u003C/p>",{"type":15,"items":349},[350],{"caption":351,"asset":352},"The sequence of A/B test steps.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-11.png",{"type":12,"text":354},"\u003Cp>Why does it take so long? Well, the people who come up with appropriate problems or test ideas often need to sell the idea to their team first. Once this is done, the test needs to be defined. This can be done in a couple of hours if the test is simple (e.g. just a configuration change). However, if the test aims to solve a new problem or requires a big new product feature, it might require several meetings with interdisciplinary teams, pre-test data analysis, and a considerable amount of documentation (e.g. new feature specification, quality assurance test scenarios, user acquisition plan, specification of new A/B test infrastructure features, etc.). Then, the test needs to be added to the work plan and may stay on hold until each required team can execute their associated tasks (e.g. creating art, developing and testing code).\u003C/p>\u003Cp>After the test is implemented, it’s time to start collecting test users and data, which can take several weeks or even months. Finally, the results of the test are analyzed and presented in order to make the final decision.\u003C/p>\u003Ch3>Implementing A/B tests\u003C/h3>\u003Cp>When it comes to implementing A/B tests, it’s helpful to communicate clear test goals to the entire team before starting to work on it. This empowers team members to contribute ideas and identify potential issues, and it decreases the likelihood of miscommunication. \u003C/p>\u003Cp>Next comes quality assurance, which ensures the tested features and data collection are implemented correctly. This is crucial because without it potential issues (e.g. bugs, biases or missing data) can go unnoticed until the test data is analyzed, incurring costs along the way and rendering your results completely useless. \u003C/p>\u003Cp>Additionally, it's crucial to consider how user experience will be handled after the test is completed. As mentioned above, this might be as simple as switching all test users to the winning variant or as complex as supporting all the test variants until the end of their lifetime.\u003C/p>\u003Cp>Another key component to efficient A/B testing is reliable test infrastructure. In our case, this includes game and test configuration dashboards, backend services for assigning users to test groups, and collecting, storing and aggregating the test data. This enables us to do simple A/B tests with minimal effort. For example, to test the impact of wardrobe item prices, we only need to add price configurations for the test groups (as shown in the figure below), select how many and which users (e.g. by country, application version, etc.) we need per test group, and press the “start test” button. Over the following days our A/B test dashboard refreshes automatically, clearly showing the impact of the price changes on all relevant game performance metrics using graphs, tables, and statistical results.\u003C/p>",{"type":15,"items":356},[357],{"caption":358,"asset":359},"Simplified example of A/B test configuration to test the impact of in-game prices.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-12.png",{"type":12,"text":361},"\u003Ch3>Collecting data\u003C/h3>\u003Cp>We’ve already discussed how many users are needed per test group, but it’s also important to consider the time frame in which they are added to the test and exactly how long you’ll be collecting their data for.\u003C/p>\u003Cp>The amount of time it takes to collect users depends on the number of required test users, whether they need to be new or existing users, and the amount of users that can be acquired each day. \u003C/p>\u003Cp>In our case, this typically takes around one to four weeks, or in some cases up to eight weeks. Even if you can acquire enough users in a single day, you may need to slow down and extend user acquisition to cover an entire seasonality period. For Outfit7, this is a one-week period, because new users who install our games during weekends have different behavior compared to those who install on a workday. Furthermore, you may want to avoid acquiring users during certain periods, such as holidays, because they can bias your sample.  Although your  effect will be the same on all test groups, it might differ from the effect you will have on users obtained during normal days.\u003C/p>\u003Cp>Second, users must keep using the designated product variant long enough to acquire enough required data. For example, if the test group affects the initial experience and its impact on the observed metric is short-lived, it may be enough to observe the users for a week. However, if users experience the change in product only after some time (e.g. we change the difficulty of levels that can only be reached after one month of play), or if the effect of the test group changes through time (e.g. user engagement with the new feature decreases over the first three weeks after their first use), or if the observed metric has spikes spaced over long time intervals (e.g. buying a monthly battle pass), you’ll need to be collecting test data over a long period of time.\u003C/p>\u003Cp>The data collected must include at least the test group id and data that enables you to compute the relevant test metrics. Additional data is often very useful. For example, data for computing your key performance indicators enables you to detect unexpected test influences; market segment data enables you to compare test effects on each market segment (e.g. country, platform, user source); and user ids enable you to analyze the impact of the test per user. In general, collecting as much data as possible (without negative impacts on user experience and with acceptable infrastructure cost) is preferable because new questions often pop up while you’re in the process of analyzing your test data.\u003C/p>\u003Cp>If multiple A/B tests are conducted on the same users, it’s important to prevent bias caused by the past or concurrent A/B tests. For example, if we conducted Test 1 with groups A and B, we must make sure that in Test 2, each new test group has an equal share of users from our Test 1 groups, as shown in the figure below. This can be easily achieved by adding a unique test id as a salt to the user id, computing a hash function from it, and assigning a range of hashes to each test group.\u003C/p>",{"type":15,"items":363},[364],{"caption":365,"asset":366},"Prevent biased test results by randomizing test group assignments.","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-13.png",{"type":12,"text":368},"\u003Ch3>Analysis\u003C/h3>\u003Cp>Some data analysis can be automated, while some manual data analysis can be done while the data is being collected. However, the final test results can only be calculated after all the required data is collected. The figure below shows our test data analysis pipeline.\u003C/p>\u003Cp>We collect users’ data in a BigQuery data warehouse, which enables easy access to data for automated and exploratory analysis, as well as efficient, cost-effective storage and processing of large amounts of data. Then, we use the dbt data transformation tool to define how data aggregates, such as the key test metrics, are computed. It enables data analysts to use SQL in order to define the data transformations and ensure data quality, centralizes data documentation and discovery, and makes complex data processing reliable and maintainable – some aggregates are the result of more than 10 consecutive transformations of data obtained from multiple sources. \u003C/p>\u003Cp>Our data transformations are run using Apache Airflow, which automatically schedules (and also retries when necessary) data transformation jobs, enables re-running selected jobs or all jobs that depend on their results (in case of bugs or when adding new aggregates for past data), and enables scalability and operability of data processing pipelines. Finally we use data processing and visualization tools to create automatic reports and share them with the team. We use Looker Studio to quickly build simple dashboards, and Posit Connect to create custom complex dashboards and reports that require access management using R code and packages, such as Shiny, ggplot2, tidyr, dplyr, purrr, rmarkdown, and knitr.\u003C/p>",{"type":15,"items":370},[371],{"caption":372,"asset":373},"A/B test data pipeline and data analysis tools","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/ab-testing-14.png",{"type":12,"text":375},"\u003Cp>In addition to automated analysis, we also conduct custom data analysis when testing new features, or when in depth analysis is required. The analysis code is stored in git which allows for efficient code reviews, future reference, extending the analysis, and reusing the code. Test analyst interprets the data and writes a test report (which is thoroughly peer reviewed) and finally presents the report to decision-makers. We use Atlassian Confluence to publish and share test reports, which include a short executive summary with a couple of main plots, links to test specification and configurations, over 60 standard plots exported from the A/B test dashboard and additional custom plots and tables – everything neatly organized as a single source of truth.\u003C/p>\u003Ch3>Decision making\u003C/h3>\u003Cp>After a test report is peer reviewed, the relevant decision can finally be made based on the results. In some cases, this may be as simple as a five minute presentation to the decision-makers who all agree to a simple outcome (e.g. “We all agree with the proposed decision to switch all users to the winning group.”). On the other hand, the results may lead to further questions and the need for additional data or analysis. Sometimes, the broader consequences of switching to the new variant are unclear or cannot be measured. In these cases,  the decision will be more complex, taking into account the knowledge and experience of the team, as well as the broader business position. Sometimes you will even arrive at the outcome that you need to return to one of the previous A/B test steps to gain more insight and arrive at a better or safer decision.\u003C/p>\u003Ch3>Conclusion\u003C/h3>\u003Cp>A/B testing can either be a scientifically correct and highly effective tool for optimizing products or it can be a big waste of time and money. The outcomes depend mainly on the people and the process of execution. I hope that this blog post inspires you to conduct your own A/B tests, and that the insights shared help you get the most value possible out of them.\u003C/p>","Rok Piltaver, Senior Data Scientist",[378],{"id":379,"title":380,"slug":381},"tech_categories::analytics","Analytics","analytics",{"title":383,"copy":22,"backgroundImage":384,"cta":385},"Lessons learned from 10 years of A/B testing - Part 2","https://cdn-o7.o7web.com/assets/4_blog_news/tech_blog/web_card_web_card_aug24_14_techb-abtest2.jpg",{"title":32,"type":33,"href":386},"/blog/tech/lessons-learned-from-10-years-of-a-b-testing-part-2","lessons-learned-from-10-years-of-a-b-testing-part-2",{"title":389,"description":7,"canonical":390,"robots":7},"Lessons learned from 10 years of A/B testing - Part 2 | Outfit7","http://localhost/blog/tech/lessons-learned-from-10-years-of-a-b-testing-part-2",{"open_graph":40,"twitter":40,"site_name":41,"title":383,"description":392,"image":393},"How we do A/B testing at Outfit7","https://cdn-o7.o7web.com/img/asset/YXNzZXRzLzRfYmxvZ19uZXdzL3RlY2hfYmxvZy9saXBvc3RfLXNlcDI0XzI1X3RlY2hiLWFidGVzdDIuanBn?w=1200&h=630&q=70&fit=crop&s=9c5037b8b2135d37d765c2a23943e7a0","content:tech:c8e19920-6583-42f3-bcff-a71e063f4030.json","tech/c8e19920-6583-42f3-bcff-a71e063f4030.json","tech/c8e19920-6583-42f3-bcff-a71e063f4030",13,1764606561180]