In this episode, Austin Parker, Principal Developer Advocate at Lightstep talks about the OpenTelemetry Framework, which is an observability framework for cloud-native software and a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior.

In this episode, Puneet Lath, Director of Research and Development at Expensify, talks about the unique way Expensify is using open source with their products by not just open-sourcing software tools but also open-sourcing the front end of the product itself. They are rebuilding their products on React Native to be fully cross-platform and doing so in an open-source manner. All code is public, and anyone can see it and contribute to it.

In this episode, Steve Pereira—the founder of Visible—talks about how his unique approach to mapping helps customers get products out of the door fast and efficiently. Maps flow to build alignment, clarity, and confidence, and Steve focuses on two primary areas to drive exceptional business outcomes: Flow and value. Value stream thinking and methodologies, augmented by continuous improvement and performance.

Data is at the center of everything we do. Yet, how can we trust it in a world where more "organic" food is consumed than produced? In this episode, Jason Kelley—the Global General Manager for Blockchain Services at IBM—talks about how data trust and transparency are applied to COVID-19 vaccines.

"Java has done an excellent job at ruining types for everyone for quite a while—explains Marcel after describing the tech pub scene in London—but it's important to know there's more than one kind of type system." Along with Shane, they outline what's exciting about ReasonML and their experience with new languages and tools around types.

SPECIAL GUEST:

• Marcel Cutts | @marcelcutts: Founder of Asgard Tech, ReasonML London Organizer
• Shane Wilson | @wilsonashane: Founder of Peyk Inc.

SHOW NOTES

00:38 - Reason

• Twitter: @reasonml
• Discord

03:25 - BuckleScript

• OCaml

06:01 - Reason + BuckleScript

• TypeScript

16:07 - Reason: Interoperation & Adoption

• ClojureScript
• Toolchain
• Syntax

30:00 - Operating at the Compiler Level vs the Run-Time Level

• ppx (pre-processor extension)

34:29 - Last thoughts on, and why use Reason?

• Flow

44:43 - repkgs

• knit
• Lerna

Monorepos are the new muse of library maintainers, but what happens when your project grows past 100 packages in the same repo? What about thousands? Rush.js was created for those cases, and Pete—who started the project while working at Microsoft—is here to tell us about it.

SPECIAL GUEST:

Pete Gonzalez | @octogonz

During the day, Pete works at HBO in Seattle on their streaming media apps. Prior to that, he was at Microsoft for 9 years, and before that, he worked at various consulting companies. A long time ago he was a cofounder of Ratloop, a small company that makes video games.

SHOW NOTES

01:24 - Rush.js: What is it and what is it for?

• Rush on GitHub

04:47 - Problems with Managing Large Codebases

• Rush Stack: provides reusable tech for running large scale monorepos for the web

07:22 - How does Rush provide a solution for build orchestration?

13:34 - Rush Stack Opinion: How to Lint, Bundle, etc.

16:53 - Using Rush Stack: Getting Started

• The pmpm Package Manager
• Yarn Plug’n’Play

24:27 - Getting Technical About Versions

• Phantom Dependencies
• Doppelgangers
• Pure Dependencies

32:47 - Thoughts on Monorepos

36:30 - Getting Started (Cont’d) + Efficient TypeScript Compellation

43:28 - Does Rush have a size limit? Is it for bigger or smaller projects? Both?

44:34 - Using pieces of Rush in non-Rush projects?

In this episode, Charles and Taras discuss "big ideas" and all the things they hope to accomplish at The Frontside over the next decade.

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at [email protected]. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. Today, we're going to talk about big ideas and the future of Frontside.

TARAS: Yeah, starting with is Frontside is good idea?

CHARLES: No, we're going to just talk about how do you know that an idea is good? We've touched on it a couple of times before. Like how do you, how do you go about validating a big idea, how do you discover a big idea? What do you do?

TARAS: And then, even when you have big ideas like big tests, what does that mean for the world? How do you make a big idea an idea that a lot of people like and agree with and actually use on day to day.

CHARLES: Yeah. It turns out that it's not easy. There's a lot of work involved with that. A lot of it crystallized around the conversations we're having about what exactly is big test and recognizing that big test isn't really a code base. It's not a toolkit. Even though it does has aspects of those things, it really is an idea. It's an approach. It's a way of going about your business, right?

TARAS: Yeah. Especially when you put big test functionality in place, when you start doing big testing and then you put together things like using Mocha and Karma, big tests in that kind of test suite is really just like interactors and the idea of big testing. There's nothing else. All the interactors do is just give you an easy way to create composable like [inaudible] objects, so you don't have to write -- you have the components but you don't have to write selectors for each element in the component, especially if gets composed. But that's like a very small functionality that does a very specific thing. But big test itself, it takes a lot of work to actually -- we had this firsthand experience on the project we're working on right now. We are essentially introducing like Ember's acceptance testing but for React in a react project and having to explain to people what is it about this that actually makes it a really good idea and having people in the React world see that this is actually a really good idea. It's kind of incredible. When you actually try to sell something to somebody and convince somebody that this is a good idea is when you realize like how inadequate your understanding of the idea really is. You really have to start to break it down and understand what is it about this that is a really big idea.

CHARLES: Yeah, I completely agree 100% because to be clear, we've actually been doing this now for two years almost. So, this is not the first React project where we've put these ideas in place. But I think in prior examples, we just kind of moved in and it's like we're going to do this because this is what we do. And we have firsthand knowledge of this working because we've operated in this community where this is just taken on faith that this is the way you go about your business. You have a very robust acceptance test suite. And because of that, you can experience incredible things. When you and I were talking before the show, we were kind of commenting on inside the Ember community, you can do impossible things because of the testing framework. You can upgrade from Ember 1 to Ember 3 which is a completely and totally separate framework, basically. You're completely and totally changing the underlying architecture of your application. You can do it in a deterministic way and that's actually incredible.

TARAS: And what's interesting too is that the React core team kind of hinted a book at this also in their blog post about fiber or moving to fiber because one of the things that they talked about there is that knowing how the system is supposed to behave on the outside allowed them to change the internals of the free app framework, specifically about test suite for the React framework, but it allowed them to change the internals of the framework because they were testing kind of on the outside. The system kind of is a black box and that allowed them to change the internals and the test suite essentially stayed the same. So this idea of acceptance testing your thing is really fundamental to how Ember community operates. But other communities have this as a big idea as well. It's just applied in different areas.

CHARLES: Right. And so applied to your actual application, this is something that's accepted in one place, but it's not an accepted practice in other places. But you can make your argument one of two ways and say, "I have experienced this and it's awesome. And you can do architectural upgrades if you follow the patterns laid out by this idea." And that works if you have a very, very high, high trust relationship, but you don't always have that, nor should you. Not everybody is going to be trusting you out of the box. So, you're going to have to lay out the arguments and really be able to illustrate conceptually practically how this is a good idea just so that people will actually give it a try.

TARAS: And it takes a lot. One of the reasons why it was easier for us to introduce big testing to the current project we're working on is because we were able to write, like we've done the implementation for this in a previous project. So when we were convincing people this is a good idea, we're like, "Look, your test suite can be really good. It's going to be really fast. And look, we've done it before." So the actual process of convincing people that a big idea is a really good idea is actually kind of a complicated process that requires a lot of work and partially requires experimentation. You have to actually put an implementation in place and show that you're going to have to build up on your successes to be able to get to a point where you can convince people that this is actually a really good idea. People who have not heard about this idea, and especially people who might have counter, like they have people of authority in their community that have counter views. For example, quite often when it comes to big tests, when you bring up big tests, people will reference a blog post by a Google engineer that talks about how functional testing or acceptance testing is terrible. And for a lot of people, a Google engineer means a lot and the person makes really good points. But that's not the complete idea. The complete idea is not just about having an acceptance test suite. It's a certain kind of acceptance test suite. It's an acceptance test suite that mocks out at boundaries so you don't make API requests to the server, you make an API request to a [inaudible] server using something like Mirage or whatever that might be. So, the big idea, it has like new ones that makes it functional, but getting people who are completely unaware, who don't necessarily look up to you as an authority to believe you like, "Yes, that actually sounds like a really good idea." It is not a trivial task.

CHARLES: No, it's not. Because the first time you try and explain it, you're arguing based on your own assumptions. So, you're coming to it safe in the knowledge that this is a really, really good idea based on your firsthand knowledge. But that means you're assuming a lot of things. You're assuming a lot of context that you have that someone else doesn't and they're going to be asking questions. Why this way, why this way, why this way? And so, you have to generate a framework for thinking about the entire problem in order to explain the value of the idea. And that's something that you don't get when it's just something that's accepted as a practice.

TARAS: I think simulation is actually a really good example of that. If you haven't had experience with Mirage, if you don't know what having a configurable server in your tests does for you, you will probably not realize that similar ideas apply to, for example, Bluetooth that when you're writing tests for your Bluetooth devices, or you're writing tests for an application that interacts with Bluetooth devices, you actually want to have a simulation there for Bluetooth so that you can configure it in a kind of similar way to the way you would configure a Mirage for a specific test. You want to be able to say, "This Bluetooth device is going to exist," or, "I have these kinds of Bluetooth devices around me, they have the following attributes. They might disconnect after a little while." There's all kinds of scenarios that you want to be able to set up so that you can see how your application is going to respond. But if you haven't seen a simulation with something like Mirage, you're going to be going like, "I don't know what the hell why would this be helpful."

CHARLES: It seems like lot of work. One of the things that we've been working on, as I said, is trying to come up with a framework for thinking about why this is a good idea because we can't just assume it. It's not common knowledge. For example, one of the things that we've been developing over the last year and more recently in the last few months is trying to understand what makes a test valuable. At its essence, what are the measures that you can hold up to a test and say this test has X value. Obviously, something like that is very, very difficult to quantify. But if you can show that and you say, "This test has these quantities and this test has these quantities," then we can actually measure them. Then it's going to allow people to accept it a lot more readily and try it a lot more readily. So, the ideas that we're playing with right now is that you kind of have to evaluate a test on one on speed, tests that are fast, have an intrinsic value, or rather test that are slow. The upside that you gained from the test is very quickly bled away or offset if the test is slow. So, I can have a very comprehensive test that tests a lot of high value stuff. But if it takes three days to run, it's going to be basically worthless. Another axis on what you can evaluate is in terms of coverage. I'm not talking about coverage of lines of code. I'm talking about use cases and units of assemblage. So, there's the module. Those modules are then stitched together into components. Those components are stitched together into applications. Those applications are downloaded onto browsers. And I would consider it a different unit of assemblage. Your application running on Firefox is a different assemblage than your application running on Chrome, is a different assemblage than your application running. You have this access, which is the coverage of your unit of assemblage. That's another way that you can evaluate your tests. So if I have a test that runs only on Node in a simulated dom, there's a cap, there's an absolute cap on the value of that test and it cannot rise above a certain point. And the other thing, another access that we've identified is isolate ability, an ability to run a test. So if I have a test suite comprised of 1500 tests, but if one fails in the middle, I have to restart the test from the beginning. That's going to decrease the value. Maybe it's related to speed, being able to run the tests without having to install a bunch of different dependencies. So that's another access. And so trying to really understand the variables there, that's something you have to be very systematic about thinking about the tests so that you can actually take your idea and explain it to someone who's not coming at it from first principles. Imagine you have to explain an if statement to somebody who's never programmed with an if statement before.

TARAS: That is going to be very difficult.

CHARLES: It's going to be difficult, right?

TARAS: Yeah. In general, it's very challenging to go from an experience that somebody had, like overriding somebody's experience conveying your own personal experience is very difficult. And getting someone to experience something is very difficult. And so that's what I think a lot of the work that we've been doing over the last little while has been breaking down these problems into a way of understanding them so that we can actually explain why these things are important. Like what we've had to do this recently with a Bluetooth work that we've been doing. We have a partner that's implementing a Bluetooth abstraction for mobile devices and trying to convey to them the value of being able to simulate devices. That's something that we saw with Mirage. We knew that being able to simulate devices in tests in different environments is extremely valuable. We know this firsthand from our experience. But trying to justify to them why we think this is important and why they should rejig all of their thinking about how they're going to architect this middle layer between the native APIs and the application APIs, why they should rejig this layer to make it so that it will allow for simulation, like convincing very technical, very knowledgeable and experienced people that these ideas are important, it requires kind of fundamental understanding of the value that they provide.

And I think this touches on what really I think Frontside is going to be doing going forward is creating conditions for this kind of thought to be cultivated, to be able to create business environment where our clients gain benefit from this kind of very deep insight, insight that transcends the source of those ideas. Because there are certain ideas that are really fundamental and they're beautiful ideas. For example, in React world, a functional component is a really beautiful idea. It's a really simple concept. And kind of going along along with what you're saying about this kind of fundamental ideas, they will drive you to the next point you couldn't even imagine. I think the work that the React core team has done -- they've set out this functional component as a primitive, but it wasn't possible for a long time to really make the functional component a reality until they've gone through the process of actually understanding what connects all the dots so that you could eventually get to a point where like, "Oh, actually this [inaudible] API is a way for us to enable the fundamental concept of having a component." Not only is it simple, but it actually works. You can write a performance application using this fundamental concept. So, the work that we're going to be doing is first of all, making it possible to have conversations at this level. So, people that we're going to be working with Frontside, clients who we work with, companies that work with us, who partner with us, it's going to be all to support creating this environment where we can have this kind of ideas. Then being able to extract these great ideas from the frameworks and actually making them available across frameworks. You don't have to be locked into a specific community to be able to benefit from that.

One of the first steps we're working on now is now that we have an understanding about big tests, we believe this is a good idea, we have ways to justify why it's a good idea. We have clients who have benefited from this good idea. Now we're in position to fill in the gaps that are missing from big tests, from being something that could be used in any framework. We want it to be really easy for people to be able to say, "I really love acceptance testing, but I have to work in a React project or I have to work in a Vue project, I have to work in an Angular projects." It's like, "It's no problem. I have big tests. Big tests is going to give me what I want." The big tests, the idea, but it's going to come with all the little pieces that you need to be able to assemble the functional test suite that is going to give you the benefits that Ember's acceptance test suite provides for Ember projects.

CHARLES: I mean, why should you have to compromise on these things. If it is a good idea and it is a fundamental idea, whether it's how you manage concurrent processes, whether it's how you manage testing, whether it's how you manage routing, why should you have to compromise? Why should you have to say, "You know what? I love..." I don't know, what's a comparable system? "I love air conditioning." Why should I have to go into a car that doesn't have air conditioning? Because every single car has a different air conditioning system, or every single house has a different air conditioning system. I'm showing the fact that I live in Texas here by using this example. But we've developed air conditioning systems that are modular so that they can be snapped on to pretty much any house and you don't have to build a custom thing for the circulation and refrigeration of air every single time or have it just be like it's part of the house. Or we have this wood that ships with ducks. It's like, no, we want to separate out that system.

TARAS: You don't have to commit to living in a specific area to get the benefit of being comfortable. You don't have to give up the comfort for specific areas. The good idea of air conditioning is not restricted to just one area. You can actually experience it wherever you go and have it be available wherever you would go. You have to be able to, like if you're moving to a new house, you can install air conditioner and now you're going to have like cool air. That's the kind of the theme I think of what we're going to be doing. But there's so much work to do because we're kind of, I would say, probably 30 or 40% into having -- if we were to look at like big tests as a big idea is used and is well known in our industry, we're probably maybe 5% on that. If we take into consideration what it takes for us to be able to convey what it takes to convince people is a good idea. I think we're, if you add to it, we'll probably add another 15%. So the next step is actually creating some tooling around it that would make it really easy for people to consume. Because right now, what's really unfortunate is that you have big tests that is available. If we set it up in the React project, doing it with create-react-app right now is kind of difficult. So we're going to make that easier. But then companies still need to make a choice between the ergonomics [inaudible] Cypress or the speed and the comfort and the control that you have with big tests. We have to basically eliminate the need to make that choice by making the tooling that you get in Cypress and making that tooling available for big test projects.

So that is going to kind of bring us up to maybe 60%. And then the remaining part is taking the software that enables the big idea and then making the software work across all different frameworks so that it's really easy to install. On Angular, it's just going to be add a plugin and it gives you big testing. You don't have to rely on their unit testing or you don't have to rely on the end to end testing as using Selenium. You just have big tests and it works just as well as it works and React and just as well as it works in Ember. So you can get the benefits of that. Going through the whole process and then bringing into the world this way, we have a lot of work to do because every part of the architecture of building modern frontend applications requires this level of effort. If you look at routing, routing is extremely inconsistent across frameworks. There are different opinions for every framework. There are different opinions of how to implement it. And that's problematic both on an individual level and a specific project level because it's so hard to know what you should use. And in many cases, it's not complete. What constitutes a routing system on React looks different than what would be in Ember, and it looks different than what it is in Angular. And each one of them has their own trade-offs. So if you find yourself in a situation where you've been using React and now you have to use Angular, you have a steep learning curve. But this also has an interesting effect of like, "Well, we have a world now where micro frontends are a thing." There are really good reasons why companies might want to use multiple frameworks for single platform because different frameworks have different benefits and different teams based on their location might prefer a different framework. There's lots of different reasons why companies choose and we want developers to be able to choose specific framework. But how do you do that when you need to have basically micro frontend architecture, you need to provide an SDK that provides a consistent user experience like -- you need to provide an SDK that provides a consistent developer experience, but then that developer experience because it's developer experience, needs to enable consistent user experience regardless of what framework the team decides to use. So now, routing needs to be more robust. Routing without having a strong concurrency primitives is going to be very limited and it's not going to be as portable across frameworks as one that actually has very robust concurrency primitives.

So now we'll start looking at concurrency. We've got concurrency primitives that are different for each framework. We've got Ember concurrency, we have a Redux-sagas. We have observables. React is introducing suspense and their hooks. And now each one of those things is different. So now, we need a concurrency primitive that is framework agnostic that is externalized, that we can consume to build stuff. Now that's a piece of the routing system. There's other things like state machines. State machines are an important part. If you look at the authentication system, the core of an authentication system is a state machine. How do you express a state machine in a way that is framework agnostic, in a way that works in every framework. That's an area that we need to explore. So, there's a lot of work for us to do to take these big ideas and externalize them from the framework so that they can be consumed by frameworks. And the applications that use this frameworks, that is a lot of work. And that's essentially what Frontside is setting out to do is hold these good ideas and extract the pieces that are actually core to these ideas. And then make them available independently and then see what that will provide. In a world where we have all of these pieces in place and we have kind of the granular little pieces that we need to be able to have a really strong concurrency primitive, we have a great routing system that's using this concurrency primitives. We have a state machine mechanism that can work with any framework, has very comfortable APIs. We have all these pieces. What kind of collaboration will that allow? What is the world that that will create? I think for people who are in the Ember community, we saw firsthand what it means for us to stand the shoulders of giants. We have wonderful, brilliant people creating really awesome tools. Now, if that same level of collaboration was happening in a way that was framework agnostic, but we were all standing on shoulders of giants and we were helping each other create something like this, what kind of world would that create? That's what Frontside is setting out to find out.

CHARLES: Yeah, and it's going to be fantastic. I should say it's going to be really, really fun. It's going to be challenging. Like you said, there is a lot of work to do.

TARAS: A lot of conversations to have. I mean, I think it's difficult to have these conversations because we all have good reasons for thinking certain things and a lot of times it requires understanding each other to find the place. Core teams do this. It's getting everybody to align on where we want to go, provide leadership like doing all of that work and doing it in a way that is actually possible. Because to create an environment with this kind of research that is happening, it requires a lot of money. Charles and I cannot do this. A small group of people probably cannot do this either because we need real use cases. We need real projects to make sure that these ideas are not dream implementations or just like vaporware. We need the pressure of real projects, real clients' relationships to create software that is going to make a huge difference for their developers. So we need those relationships with clients so that we can actually create the conditions for this kind of thing to emerge. That needs to also happen for us to make this kind of a situation possible. So, there is a lot of work to do but it's a really worthwhile cause.

CHARLES: And I think that it's one of the reasons that we start with testing as the most important use case because it's something that you don't want to be just saying, "Hey, you want to come help us fund research to make everything better?" There is real pressure, there's financial pressure and there's a duty to make sure that if a project has a deadline, that deadline is met. You want to come in at cost and on time. That's the goal of every project. And you want to try and strive to do your best to achieve that. And testing is an area where the research almost always pays off. I mean, where it has clearly. You can invest a lot of money and get a lot of reward in return because you're losing so much money by not testing or by focusing your investment in testing on lower value targets.

TARAS: For people that are familiar with acceptance testing, I think that just goes like a no brainer. Acceptance testing just enables you that. So by putting big test as a first kind of objective, but putting acceptance testing as a first kind of milestone, it sets us up to benefit in the way that Ember community has benefited where the acceptance test now became a mechanism that made it possible for the framework to change. And so, we're kind of setting up the same conditions here. When you start working with us or with any of the companies that work with us, the first benefit you get is you get the acceptance testing. Now, developers are more productive because they know that they have a certainty that if something broke, they will know. If they broke something, they will know. But what that also gives us is a way of introducing change in a systematic way. So, if we find that we have a new routing mechanism, we can start to refactor our React router setup to use this new routing mechanism and we can do it safely because we have our acceptance testing in place. So in many ways, it's kind of like the first step. And it's actually a great first step for business and is a great first step for tech technology. That's kind of a really awesome overlap.

CHARLES: In the podcast with Dylan Collins, we talked about this is like step one is make experimentation cheap because experimentation can be wildly expensive and you can throw away a lot of money if you're just experimenting without constraints. And there is nothing like a great application/acceptance test suite that puts those constraints in a completely and totally unambiguous way. So I think that's actually a great summary of our next decade.

TARAS: Yeah, I think so too. I think it's a good to show you the perfect next step. And I think this is going to show us the perfect next future though we can't really imagine it right now. But I do know one thing, this future is going to include a lot of people. Fundamentally, what we're setting out to do is a big idea. And so, big ideas are brought to the world by a lot of people. And I think one of the things that's important is that from day one, we include everyone in the process. For anyone who's interested in these big ideas, we want to make it possible for you to participate in. So that's the reason why Frontside is going to have an RFC. Like on our Frontside website, we're actually going to start writing and publishing RFCs. The first RFC that we're going to have is going to be for big tests, which is to lay out why big test is a big good idea for frontend for our industry. And as we start talking to people, there's actually lots of applications beyond frontend. So, we're going to focus on writing these RFCs and sharing them with the world so that we can include as many people in it as possible so everyone can participate. And ultimately, hopefully, that will have an influence on our industry and actually allow us to leverage these big ideas in a way that is not tied to any particular source of these brilliant ideas.

CHARLES: Yeah, hat tip to the Ember community for the whole RFC thing, which I guess they didn't invent either.

TARAS: Yeah, they borrowed it also, sorry. But I mean, it's been adopted everywhere. And I think it is a really interesting way of making people part of your process and making it our process as our meaning, like everyone who works with us. Whether you work for Frontside, that there's a client of Frontside or you work with the company that is partner with Frontside, whatever that might look like, we want to start to create places where we can be having these conversations and together surfacing these great ideas and making them part of a tool sets.

CHARLES: I think that's a perfect way to wrap up. Like I said, you pretty much described our next decade. Maybe if we get more people, it's going to be significantly less and we'll move onto the next thing even more rapidly. I am fully pumped. I'm ready to see this. There's a lot of work to do and I feel great about getting to it.

TARAS: I'm very excited about all the work that we're going to be doing, so let's get to it.

CHARLES: Yup. All right. We'll see everybody next time.

Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at [email protected]. Thanks. See you next time.

In this episode, Charles and Taras discuss "transparent development" and why it's not only beneficial to development teams, but to their clients as well.

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at [email protected]. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build them right.

It's been a long summer and we're back. I'm actually back living in Austin, Texas again. I think there wasn't too much margin in terms of time to record anything or do much else besides kind of hang on for survival. We've been really, really busy over the last couple of months, especially on the professional side. Frontside has been doing some pretty extraordinary things, some pretty interesting things. So, we've got a lot to chew on, a lot to talk about.

TARAS: There's so much stuff to talk about and it's hard to know where to start.

CHARLES: So, we'll be a little bit rambly, a little bit focused. We'll call it fambly. But I think one of the key points that is crystallized in our minds, I would say over this summer, is something that binds the way that we work together. Every once in a while, you do some work, you do some work, you do some work, and then all of a sudden you realize that a theme, there's something thematic to that work and it bubbles up to the surface and you kind of organically perceive an abstraction over the way that you work. I think we've hit that. I hit one of those points at least because one of the things that's very important for us is -- and if you know us, this is things that we talk about, things that we work on -- we will go into a project and set up the deployment on the very first day. Make sure that there is an entire pipeline, making sure that there is a test suite, making sure that there are preview applications. And this is kind of the mode that we've been working in, I mean, for years and years and years. And where you say like if what it takes is spending the first month of a project setting up your entire delivery and showcasing pipeline then that's the most important work, inverting the order and saying that has to really come before any code can come before it. And I don't know that we've ever had like a kind of unifying theme for all of those practices. I mean, we've talked about it in terms of saving money, in terms of ensuring quality, in terms of making sure that something is good for five or 10 years, like this is the way to do it. And I think those are definitely the outcomes that we're looking for. But I think we've kind of identified what the actual mode is for all of that. Is that fair to say?

TARAS: Yeah, I think one of the things I've always thought about for a long time is the context within which decisions are made because it's not always easy. And it's sometimes really difficult to really give it a name, like getting to a point where you have really clear understanding of what is it that is guiding all of your actions. What is it that's making you do things? Like why do we put a month of work before we even start doing any work? Why do we put this in our contract? Why do we have a conversation with every client and say, "Look, before we start doing anything, we're going to put CI in place." Why are we blocking our business on doing this piece? It's actually kind of crazy that from a business perspective, it's a little bit crazy that you be like, "Oh, so you're willing to lose a client because the client doesn't want you to set up a CI process?" Or in a case of many clients, it's like you're not willing to accept -- the client is going to say, "We want to use Jenkins." And what we've done in the past, in almost every engagement, we're like, "Actually, no. We're not going to use Jenkins because we know that it's going to take so long for you to put Jenkins in place. By the time that we finish the project, you're probably still not going to have it in place. That means that we're not going to be able to rely on our CI process and we're not going to be able to rely on testing until you're finished." We're not going to have any of those things while we're doing development. But why are we doing all this stuff? It was actually not really apparent until very recently because they didn't really had a name to describe what is it about this tooling and all of these things that makes why is it so important to us. I think that's what kind of crystallized. And the way that I know that it's crystallized because now that we're talking to our clients about it, our clients are taking on to picking up the language. We don't have to convince people that this is a value. It just comes out of their mouth. Like it actually comes out of their mouth as a solution to completely unrelated problems, but they recognize how this particular thing is actually a solution in that particular circumstance as well even though it's not something Frontside sold in that particular situation. Do you want to announce what it actually is?

CHARLES: Sure. Drum roll please [makes drum roll sound]. Not to get too hokey, but it's something that we're calling Transparent Development. What it means is having radical transparency throughout the entire development process, from the planning to the design, to the actual coding and to the releases. Everything about your process. The measure by which you can evaluate it is how transparent is this process to not just developers but other stakeholders, designers or people who are very developer adjacent, engineering managers all the way up to C level executives. How transparent is your development? And one of the ways that we sell this, because I think as we talk about how we arrived at this concept, we can see how this practice actually is a mode of thinking that guides you towards each one of these individual practice. It guides you towards continuous integration. It guides you towards testing. It guides you towards the continuous deployment. It guides you towards the continuous release in preview. I think the most important thing is that it's guided us, by capturing this concept, it's guided us to adopt new practices, which we did not have before. That's where the proof is in the pudding is if you can take an idea and it shows you things that you hadn't previously thought before.

I think there's a fantastic example. I actually saw it at Clojure/conj in 2016, there was a talk on juggling. And one of the things that they talked about was up until I think it was the early 80's or maybe it was the early 60's, the state of juggling was you knew a bunch of tricks and you practice the tricks and you get these hard tricks. And that was what juggling was, is you practice these things. It was very satisfying and it had been like that for several millennia. But these guys in the Physics department were juggling enthusiasts and I don't know how the conversation came about, you'd have to watch the talk. It's really a great talk. But what they do is they make a writing system, a nomenclature system for systematizing juggling tricks, so they can describe any juggling trick with this abstract notation. And the surprising outcome, or not so surprising outcome, is that by then, once you have it in the notation, you can manipulate the notation to reveal new tricks that nobody had thought of before. But you're like, "Ah, by capturing the timing and the height and the hand and we can actually understand the fundamental concept, and so now we can recombine it in ways that weren't seen before." That actually opened up, I think an order of magnitude of new tricks that people just had not conceived of before because they did not know that they existed.

And so, I think that's really, as an abstract concept, is a great yardstick by which to measure any idea. Yes, this idea very neatly explains the phenomenon with which I'm already familiar, but does the idea guide me towards things with which I have no concept of their existence? But because the idea predicts their existence, I know it must be there and I know where to look for it. And aha, there it is. It's like shining a light. And so I think that that's kind of the proof in the pudding. So that's a little bit of a tangent, but I think that's why we're so excited about this. And I think it's why we think it's a good idea.

TARAS: Yes. So what's also been interesting for me is how universal it is. Because the question of is this transparent enough? That question could be actually asked in many cases. What's been interesting for me is that asking that question in different contexts that I didn't expect actually yielded better outcome. At the end of the day, I think that a test for any idea is like, is it something that can help you more frequently than not? Like is it actually leading you? Does applying this pattern, does it increase the chances of success? And that's one of the things that we've seen, thinking about just practices that we're putting into place and quite asking are they transparent enough? Is this transparent enough? It's actually been really effective. Do you want to talk about some of the things that we've put in place in regards to transparency? Like what it actually looks like?

CHARLES: Yeah. I think this originally started when we were setting up a CI pipeline for a native application which is not something that we've typically done in the past. Over the last, I would say, 10 years, most of our work has been on the web. And so, when we're asked to essentially take responsibility for how a native application is going to be delivered, one of the first things that we asked kind of out of habit and out of just the way that we operate is how are we going to deliver this? How are we going to test it? How are we going to integrate it? All the things that we've just talked about is something that we have to do naturally. But because this is not very -- like continuous integration and build is very prevalent on the web. I think that testing still has a lot of progress on the web, but it's far more prevalent than it is in other communities, certainly the native community. So when we started spending a month setting up continuous integration an integration test suite, spending time working on simulators so that we could simulate Bluetooth, having an automated process with which we could ship to the App Store, all of these things kind of existed as one-offs in the native development community. There are a lot of native developers who do these things. But because it's not as prevalent and because it was new to us, it caused a lot of self reflection both on why is it that we feel compelled to do this. And also we had to express this, we had to really justify this work to existing native development teams and existing stakeholders who were responsible for the outcomes of these native development teams. So, there was this period of self reflection, like we had to write down and be transparent about why we were doing this.

TARAS: Yeah. We had to describe that in SoWs. We actually had really long write ups about like what it is that we're setting up. And for a while, it was, people I think would read these SoWs and I think they would get the what's of what we're actually going to be putting into place. But it wasn't until we actually put it into place and we've seen a few like really big wins with the setup -- one of the first ones was the setting up preview apps where preview apps in -- the web are pretty straightforward because we've got Netlify that just kind of gives it to you easily.

CHARLES: Netlify and Heroku. It's very common.

TARAS: Yeah, you activate it, it's there. But on the mobile side, it's quite a different story because you can't just spin up a mobile device that is available through the web. It's something kind of very special. And so we did find a service called Appetize that does this. And so we hooked up the CI pipeline to show preview apps in pull requests. So for every pull request, you could see specifically what was introduced in our pull request without having to pull down source code, like compile it. You could just click a link and you see a MVC stream of a mobile device and that application running on a mobile device. So the setup took a little bit of time. But once we actually put it in place and we showed it to our clients, one of the things that we noticed is that it became a topic of conversation. Like, "Oh, preview apps are amazing." "This is so cool." "Preview apps are really great." And I think in some ways, it actually surprised us because we knew that they were great, but I think it was one of the first times that we encountered a situation where we would show something to a client and they just loved it. And it wasn't an app feature. It was a CI feature. It was part of a development process.

CHARLES: Right. So, the question is then why was this so revelatory? Why was it so inspiring to them? And I think that the reason is that even if we have an agile process and we're on two week iterations, one week iterations, whatever, there's still a macroscopic waterfall process going on because essentially, your business people, your design people, maybe some of your engineering people are involved at the very front of the conversation. And there's a lot of talking and everybody's on the same page. And then we start introducing coding cycles. And like I said, even if we're working on two week iterations and we're "agile", the only feedback that you actually have, whether something is working, is if the coder says it's done. "I'm Done with this feature. I'm on to the next feature for the next two weeks." And after that two weeks, it's like, "I'm done with this feature. I'm on to the next feature." From the initial design, you have the expectation about what's going on in the non-technical stakeholders minds. They have this expectation. And then they hope that through the process of this agile iterative development cycles, they will get the outcome that satisfies that hope. But they're not able to actually perceive and put their hands on it. It's only the engineers and maybe some really tech savvy engineering managers who can actually perceive it. And so they're getting information secondhand. "Hey, We've got authentication working and we can see this screen and that screen." And, "Hey, it works on iOS now." "I have some fix ups that I need to do on Android." So, maybe they're consuming all of their information through standups or something like that, which is better than nothing. That is a level of transparency. But the problem is then you get to actually releasing the app or whether it's on the web, whether it's on native, but this is really a problem on native. You get to where you actually release the app and then everybody gets to perceive the way the app as it actually is. So you have this expectation and this hope that was set maybe months prior and it just comes absolutely careening into reality in an instant, the very first moment that you open the app when it's been released. And if it doesn't meet that expectation, that's when you get disappointment. When expectations are out of sync and grossly out of sync with reality, even a little bit out of sync with reality, you get disappointment. As fundamental and explanation of just the phenomenon of disappointment, but it's an explanation of why disappointment happens so often on development projects. Is this kind of the expectations and hopes of what a system can be in the minds of the stakeholders? It's kind of this probability cloud that collapses to a single point in an instant.

TARAS: And that's when things really hit the proverbial fan. Now, you have the opposite. So everything that was not transparent about your development process. So everything that was hidden in the opaqueness of your development process, all of those problems, either on a product side, maybe something didn't quite get implemented the way it's supposed to. Like you actually found out two weeks or three weeks before you're supposed to release that that feature wasn't actually quite implemented right. It went through testing, but it was tested against the Jira stories that were maybe not quite written correctly. So the product people are going like, "What the hell is this? It's actually not what I signed up for. This is not what I was asking for." So, there's that part.

And then on the development side, you've got all of the little problems that you didn't really account for because you haven't been shipping to production from day one. You actually have like application not really quite working right. You didn't account as supposed to integrate with some system that is using Chorus or something you didn't account for. Like you have a third party dependency you didn't really fully understand. But because it wasn't until you actually turned it on that you actually started to talk to the thing properly, and you realize there's some mismatch that is not quite working. But now you've got everything that was not transparent about the development process, everything that was hiding in the opaque corners of your development process is now your problem for the next three weeks because you've got to fix all of these problems before you release. And that's what I think where a lot of organizations kind of find themselves in is this position where they've been operating for six months and like, "Everything is going great!" And then three months or three weeks before, you're like, "Actually, this is not really what we were supposed to do. Why did this happen?" That time is really tough.

CHARLES: Yeah. That's what we call crunch time. And it's actually something that is lot of times we think of it as inevitable, but in fact it is actually an artifact of an opaque process.

TARAS: Yeah.

CHARLES: That's the time when we have to go, everybody's like, "We're ordering pizza and Dr. Pepper and nobody's leaving for a month."

TARAS: Yeah. I think there are people that do that practice like functional testing as part of development process or acceptance testing, I think they could relate to this in some cases where if you had to set up a test suite on an application that was written without a test suite, first thing you deal with are all the problems that you didn't even know were there. And it's not until you actually start testing, like doing functional testing, not integration or unit testing where you're testing everything in isolation, but when you're perceiving the entire system as one big system and you're testing each one of those things as the user would, it's not until that point you start to notice all the weird problems that you have. Like your views are re-rendering more than you expected. You have things that are being rendered you didn't even notice because it's hard to see, because it happens so quickly. But in test, it happens at a different pace. And so, there's all these problems that you start to observe the moment that you start doing acceptance testing, but you don't see them otherwise. And so, it's the process of making something transparent that actually highlights all these problems. But for the most part, if you don't know that there are transparent options available, you actually never realize that you are having these problems until you are in crunch time.

CHARLES: Right. And what's interesting is viewed through that lens, your test suite is a tool for perception. And to provide that transparency, not necessarily something that ensures quality, but ensuring the quality is a side effect of being able to perceive bugs as they happen or perceive integration issues at the soonest possible juncture. To shine the light, so to speak, rather than to act as a filter. It's a subtle distinction, but I think it's an important one.

TARAS: About functional testing and acceptance testing. I think one of the things that I know personally from experience working with comprehensive acceptance test suites is that there is certainty that you get by expressing the behavior of the application in your tests. And I think what that certainty does is it replaces hope as opposed to having hope baked into your system where you think like you're hoping. I think for many people, they don't even perceive it as hope. They perceive it as reality. They see it as, "My application works this way." But really what's happening is there's a lot of trust that's built into that where you have to say like, "Yeah, I believe the system should work because I wrote it and I'm good. And it should not be broken." But unless you have a mechanism that actually verifies this and actually insures this is the case, you are operating in the area of dreams and hopes and wishes, and not necessarily reality. And I think that's one of the things that's different. A lot of the processes around highlighting or shining light on the opaque areas of the development process. And it's actually not even just development process. It's actually the business process of running a development organization. Shining light in those areas is then what gives you the opportunity to replace hope with real validatable truth about your circumstances.

CHARLES: And making it so that anyone can answer that question and discover that truth and discover that reality for themselves. So, generating the artifacts, putting them out there, and then letting anybody be the primary perceiver of what that artifact means in the context of the business, not just developers. And so, that kind of really explains preview apps quite neatly, doesn't it? Here we've done some work. We are proposing a change. What are the artifacts that explain the ramifications of this change? So we run the test suite. That's one of the artifacts that explains and radiates the information so that people can be their own primary source. And look at it in a developer centric, although you can tell, any old person can tell if the test suite's failing, it's not a change that we should go with. But the preview app is something we take this hypothetical change, we build it, we put it out there and now, everyone can perceive it. And so, it calibrates the perception of reality and it eliminates hope. Which is like if your development process is based on hope, you are signing yourself up for disaster. I like what you said that it implies a huge amount of trust in the development team. And you know what? If you have a cracked development team, that trust is earned and people will continually invest based on that trust. But the fundamentals are still fragile because they still can open up a crack between the expectation and the reality. And the problem is when that happens, the trust is destroyed. And it's during that crunch time, if it does happen that you lose credibility and it's not because you became a worse developer. It's not because your team is like lower performing, it's just that there was this divergence allowed to open. But then the problem is that really lowers the trust and that means that unfortunately that's going to have a negative knock on effect. And reasonably so. Because if you're an engineering manager or a product manager, you're something like this and you're losing trust in your development team and their ability to deliver what you talked about, then you're going to want to micromanage them more. The natural inclination is to try and be very defensive and interventionist and you might actually introduce a set of practices that inhibit the development cycle even further and lower the team's abilities to perform right when they need to do it the most, then you end up destroying more trust.

TARAS: Yeah, it's a spiraling effect I think because it's in the process of trying to make things better. And then you start to introduce practices. Like maybe you're going to have meetings every day reviewing outstanding stories to try to get everybody on the same page, but now you're micromanaging development team. The development team starts to resent that and now you've got this like people hating their job. It starts to get messier and dirtier and more complicated. And the root cause of that is that from day one, there was a lot of just [inaudible] about getting into it and just starting to write some code but what you didn't actually do is you didn't put in place the fundamentals of making sure that you can all observe a reality that is honest. And I think that kind of fundamental principle, it's interesting how when you actually start to kind of take this idea and when you start to think about it in different use cases, it actually tells you a lot about what's going on and you can actually use it to design new solutions.

One of the things that Frontside does, I don't know if those who've kind of worked with us before might know this or might not, but we don't do blended rates anymore. Because we don't actually, one of the challenges with blended rates is that they hide the new ones that gives you the power to choose how to solve a problem.

CHARLES: Yeah. There's a whole blog post that needs to be written on why blended rates are absolute poison for a consultancy. But this is the principle of why.

TARAS: Yeah. I think it's poison for transparent consultancy because if you want to get the benefits of transparency, you have to be transparent about your people. Because alternatively what happens is that you start off relying on your company's reputation and then there is a kind of inherent lie in the way that the price points are set up because everybody knows that there is going to be a few senior people, there's going to be a few intermediate people, a few junior people. But these exact ratios of those or who is doing what, how much people are available, all of those things are kind of hidden inside of the consulting company so that they can manage their resources internally. And so what that does is it simplifies your communication with the client. But actually what it also does is it disempowers you to have certain difficult conversations when you need the most. And you could say, "Look, for this kind of work, we don't need to have more senior people working on this." We can have someone who is junior who is at like $100 an hour, $75 an hour as opposed to being $200 or $250 an hour. We can have that person working on this and we can actually very clearly define how certain work gets solved. It requires more work. But then what it does is it creates a really strong bond of honesty and transparency between you and your clients. And it gives you a way, like now the client starts to think about you as a resource that allows them to fulfill on their obligations in a very actionable way. They can now think about how they can use you as a resource to solve their problems. They don't need a filter that will process that and try to make it work within the organization. You essentially kind of become one unit. And I think that sense of unity is the fundamental piece that keeps consulting companies and clients glued together. It's the sense of like, "We can rely on this team to give us exactly what we want when we need it, and sometimes give us what we need that we don't know we need." But that bond is there. And that bond is strong because there is no lie in that relationship. You're very transparent about who are the people that's working on it. What are they actually going to be doing? How much is this costing us?

CHARLES: It's worth calling out explicitly whether on the flip side of it is, is if you have a blended rate, which is the way that Frontside operated for, gosh, pretty much forever, is that people will naturally calibrate towards your most senior people. If I'm going to be paying $200 an hour across the board, or $150 an hour across the board, or $300 across the board, whatever the price point is, they're going to want to extract the most value for that one price point. And so, it means that they're going to expect the most senior people and become resentful if what I'm paying is $300 for a task. If I've got five senior people, it's a better deal for me. For the same price to get five senior people than two senior people to a medium level people and one junior person. And so, it has two terrible effects. One is that they don't appreciate the senior people to be like, "Hey actually, these are people with extraordinary experience, extraordinary knowledge, extraordinary capability that will kick start your part." So they are under appreciated and then they're extremely resentful of the junior people. It's like, "I'm paying the same rate for this very senior person as I am for this junior person? Get this person off my project." But if you say, "You know what, we're going to charge a fifth of the cost for this junior person and we're going to utilize them," then you're providing real value and they're appreciating it. They're like, "Oh, thank you for saving me so much money. We've got this task that does not require your most senior person. That would be a misallocation of funds. I'd be wasting money on them. But if you can charge me less and give me this junior person and they're going to do just as competent a job, but it's going to cost me a fifth of the money, then that's great. Thank you." So, it flips the conversation from 'get this god-damn junior person off my project' to 'thank you so much for bringing this person on'. It's so critical. But that's what that transparency can provide. It can totally turn a feeling of resentment into gratitude.

TARAS: What's interesting is from business perspective, you make the same amount of money. In some cases, you actually make more money. I think in that way, it's a consulting company. But that's not the important part because the amount of value that's generated from having more granular visibility into what's happening is so much greater. It's kind of like with testing where any of those things where when you start to put, when you start to shine light on these kind of opaque areas and then you start to kind of flush out the gremlins that are hiding there, what you then start to do, what you kind of discover is this opportunity to have relationships with clients that are honest. So you could say, for example, like one of the things that we've done recently is we actually have like 10-tier price point model, which allows us to to be really flexible about the kind of people that we introduce. So, there's a lot of details that go into the actual contracting negotiation. But what it does is it allows us to be very honest about the costs and work together with our clients, like actually really find a solution that's going to work really well for them. And then this is kind of a starting point when we start thinking about transparency in this kind of diverse way, you actually start to realize that there are additional benefits that you might have never been experienced before. One of the things that we found recently is that one of the initiatives that we kind of launched with one of our clients is we wanted to bring together, there's a general problem that exists in large projects, which is that if you have a really big company and you have like, let's say 20 or 30 interconnected services, your data domain, like the older data, kinds of data you work with is spread over a whole bunch of microservices spread over potentially a bunch of different development teams spread over a bunch of different locations. What usually has happened in the past is each one of those problems or the domain, the data domain has been kind of siloed into a specific application. We worked with a bank in the past and that being had for every, they had 80 countries. In each country they had 12 different industries, like insurance and mortgage and different kinds of areas of services they offered. And then for each of the country, for each of the service, they had a different application that provided that functionality. Then the next step is, let's not do that anymore because we now have something like 100, 150 apps, let's bring it all together under a single umbrella and let's create a single shared domain that we can then use. And so, a GraphQL becomes a great solution for that. But the problem is that making that change is crazy complicated because the people on the business side who understand how all the pieces fit together. On the other side, you have the developers who know where the data can come from and how to make all that real. And on the other side is there's like frontend implementers who actually build in the UIs that are consuming all these services.

On a project that we're working on right now is we're building a federated GraphQL gateway layer that is kind of connecting all these things, bringing all these things together. But the problem is that without very specific tooling to enable that kind of coming together of the frontend, the backend, the business people having coming together, creating a single point of conversation and having a single point of reference for all the different data that we have to work with and different data that is available to the gateway, without having something like that, without having that transparency in the actual data model, it is really difficult to make progress because you don't have shared terminology, you don't have shared understanding of the scope of the problem. There's a lot of dots in context that needs to be connected. And for anyone who has worked with enterprise, you know how big these problems get. And so what we've done on a project that we're working on now is we actually aimed to bring transparency to this process. What we actually did is put in place, start to build an application that brings together all of the federated services into a visualization that different parties can be involved in. And so I think one of the kind of common patterns that we see with transparency in general is that we are including people in the process, in the development process that were previously not included. So in the past, they would be involved in the process either very early on or very late in the process, but they wouldn't be involved along the way. And so what this kind of transparency practice actually does is it allows us to kind of democratize and flatten the process of creating foundations for pieces that touch many different parts of the organization. And so this tool that we created allows everyone to be involved in the process of creating the data model that spans the entire organization and then have a single point of reference that everybody can go to and have a process for contributing to it. They don't have to be a developer. There's developers who consume it. There are business people that consume it. There are data modeling people that consume it. Like there's different people parties involved. But the end result is that everyone is on the same page about what it is that they're creating. And we're seeing the same kind of response as we saw with preview apps where people who previously didn't really have an opinion on development practices or how something gets built, all of a sudden they're participating in the conversation and actually making really valuable suggestions that developers couldn't really have exposure to previously because developers often don't have the context necessary to understand why something gets implemented in a particular way.

CHARLES: Something beautiful to behold, really. And like I said, it's wonderful when a simple concept reveals things that had lay hidden before.

TARAS: Yeah. It's a very interesting lens to look at things through. How transparent is this and how can we make it more transparent? I think asking that question and answering that question is what has been kind of giving us a lot of -- it had been very helpful in understanding our challenges in the work that we do on a daily basis and also in understanding how we could actually make it better.

CHARLES: I apply this concept in action on my pull requests. I've really been focusing on trying to make sure that if you look at my pull request, before you actually look at the code, you can pretty much understand what I've done before you even look at the diff. The hallmark of a good pull request is basically if by reading the conversation, you understand what the implementation is going to be. There's not really any surprises there. It's actually hard to achieve that. Same thing with git history. Spending a lot of time trying to think like how can I provide the most transparent git history? That doesn't necessarily mean exactly the log of what happened moment to moment, day to day, but making sure that your history presents a clear story of how the application has evolved. And sometimes that involves a lot of rebasing and merging and branch management.

I think another area that has been new for us, which this has revealed those things that I just described are areas where we're kind of re-evaluating already accepted principles against a new measure, but introducing an RFC process to actually a client project where we're making architectural decisions with our developers, the client's developers, external consultants. You've got a lot of different parties, all of whom need to be on the same page about the architectural decisions that you've made. Why are we doing this this way? Why are we doing modals this way? Why are we using this style system? Why are we using routing in this way? Why are we doing testing like this? These are decisions that are usually made in an ad hoc basis to satisfy an immediate need. It's like, "Hey, we need to do state management. Let's bring in Redux or let's bring in MobX or let's bring in whatever." And you want to hire experts to help you make that best ad hoc decision? Well, not really. I mean, you want to lean on their experience to make the best decision. But having a way of recording and saying this is the rationale for a decision that we are about to make to fulfill a need. And then having a record of that and putting it down in the book so that anybody who can come later. First of all, when the discussion is happening, everybody can understand the process that's going on in the development team's head. And then afterwards and it's particularly important is someone asks a question, "Why is this thing this way?" You can point directly to an RFC. And this is something that we picked up from the Ember community, but this is something that open source projects really by their very nature have to operate in a very highly transparent manner. And so, it's no surprise that that process came from the internet and an open source project. But it's been remarkably effective, I would say, in achieving consensus and making sure that people are satisfied with decisions, especially if they come on afterwards, after they've been made.

TARAS: We actually have this particular benefit that could experience that particular benefit today where one of the other things that this RFC process and transparency with the architecture, how that kind of benefits the development organization is that a lot of times when you are knee deep in doing some implementation, that is not a time you want to be having architectural conversations. In the same way like in a big football team, they huddle up before they go on a field. You can't be talking strategy and architecture and plans while you're on the football field. You have to be ready to play. And this is one of the things that the RFC process does is it allows us to say, "Look, right now we have a process for managing architecture so that with the RFC process you can go review our accepted RFCs. You can make proposals there." And that is a different process than the process that we're involved in on a daily basis, which is writing application, using architecture where we have in place. And so that in itself can be really helpful because well intentioned people can bring up these conversations because they really are trying to solve a problem, but that might not be the best time. And so having that kind of process in place and being transparent about how architecture decisions are made allows everyone to participate and it also allows you to prioritize conversations.

CHARLES: Yeah. And that wasn't a practice that we had adopted previous to this, but it's something that seemed obvious that we should be doing. It's like, how can we make our architecture more transparent? Well, let's do this practice. So, I keep harping on this. But I think it's the hallmark of a good idea if it leads you to new practices and new tools. And we're actually thinking about adopting the RFC process for all of our internal developments, for maintaining our open source libraries.

TARAS: There is something that we've been working on that we're really excited about. So, there's a lot of stuff happening at Frontside. But one of the things that we've been doing is working on something we call the transparent node publishing process, which is something that I think we originally drew inspiration from the way NativeScript has their repo set up. But one thing that's really cool about how they have things set up is that every pull request automatically is available, like everything is available. Very quickly, a pull request is available for you to play with and you can actually put it into your application as a published version in npm and actually see exactly if that pull request is going to work for you. You don't have to jump through hoops. You don't have to clone the repo, build it locally, link it. You don't have to do any of that stuff because if you see a pull request that has something that you want but then is not available in master, there's an instruction on the pull request that tells you, "Here's how you can install this particular version from npm." And so you essentially you're publishing. Every pull request automatically gets published to npm and you can just download and install that specific version for that particular pull request in your project. That in itself I think is one of those things I suspect that is going to be talked about. It actually can alleviate a lot of problems that we have on a development processes because like the availability of the work of people who are participating in the project, there is kind of a built in barrier that we are essentially breaking down with this transparent node publishing process. And so, that's something that we're very close to to having it all on our repos and we're going to try it out and then hopefully share it with everyone on the internet.

CHARLES: I didn't know that the NativeScript did this. I thought that the idea that came from it is like how can we apply these transparency principles to the way we maintain npm packages. The entire release process should be completely transparent, so that when I make a pull request, it's available immediately in a comment. And then furthermore, even when a pull request is merged, there's no separate step of let's get someone to publish it. It's just now it's on master. Now it is available as a production release. You close the latency and you close the gap and people perceive things as they are. There is nothing like, "Oh that emerged. When do I get this?" This is something that I can't stand about using public packages is you have some issue, you find out that someone also has had this issue, they've submitted a pull request for it and then it's impossible to find if there's a version and what version actually supports this? And it's even more complex between projects that actually do backporting of fixes to other versions. So I might be on version two of a project. Version three is the most recent one, but I can't upgrade to version three because I might be dependent on some version two APIs. But I really need this fix. Well, has it been backported? I don't know. Maybe upgrading is what I have to do, but maybe downgrading. Or if I'm on the same major release version, maybe there's been 10 pull requests, but there's been no release to npm. And it can be shockingly difficult to find out if something is even publicly available. And the transparency principle comes in to, "Hey, if I see it on GitHub, if I see it there, then there's something there that I can touch and I can perceive for myself to see if my issue has been resolved or if the things work as I expect."

TARAS: I'm really excited about this. I'm really excited about this kind of clarification, this crystallization of transparency. And I'm also seeing our clients starting to apply it to solving problems within their organization as well is very inspiring.

CHARLES: Yeah, it is. It is really exciting. And honestly, I feel like we've got one of those little triangular sticks that people use to find water. I feel like we have a divination stick. And I'm really excited to see what new tools and practices it actually predicts and leads us to.

TARAS: Me too. I'm really excited to hear if anyone likes this idea. Send us a tweet and let us know what you're seeing for yourself about this because I think it's a really interesting topic and I'm sure there's going to be a lot that people can do with this idea in general.

CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside, or over just plain old email at [email protected]. Thanks and see you next time.

Rich Harris talks about Svelte and Reactivity.

Rich Harris: Graphics Editor on The New York Times investigations team.

Resources:

• Svelte

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at [email protected]. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right.

TARAS: It's actually a really nice, Rich and I'm really, really happy to have a chance to actually chat with you about this because Svelte is a really fun piece technology. In many ways, it's interesting to see our technology evolve and our industry evolve through innovation, real innovation. I think Svelte 3 has really been kind of that next thought provoking technology that kind of makes you think about different ways that we can approach problems in our space. So, really excited to chat with you about this stuff.

RICH: Well, thank you. Excited to be here.

TARAS: I think quite a lot of people know, Rich, about your history, like how you got into what you're doing now. But I'm not sure if Charles is aware, so if you could kind of give us a little bit of a lowdown on where you kind of come from in terms of your technical background and such.

RICH: Sure. I'll give you the 30-second life history. I started out as a reporter at a financial news organization. I had a Philosophy Degree and didn't know what else to do with it. So, I went into journalism. This was around the time of the great recession. And within a few weeks of me joining this company, I watched half of my colleagues get laid off and it's like, "Shit, I need to make myself more employable." And so gradually, sort of took on more and more technical responsibilities until I was writing JavaScript as part of my day job. Then from there, all these opportunities kind of opened up. And the big thing that I had in mind was building interactive pieces of journalism, data-driven, personalized, all of that sort of thing, which were being built at places like the New York Times, and The Guardian, and the BBC. That was the reason that I really wanted to get into JavaScript. And that's guided my career path ever since.

CHARLES: It's interesting that this D3 and all that did come out of journalism.

RICH: It's not a coincidence because when you're working under extreme time pressure and you're not building things with a view to maintain them over a long period of time, you just need to build something and get it shipped immediately. But it needs to be built in a way that is going to work across a whole range of devices. We've got native apps, we've got [inaudible], we've got our own website. And in order to do all that, you need to have tools that really guide you into the pit of success. And D3 is a perfect example of that. And a lot of people have come into JavaScript through D3.

CHARLES: And so, are you still working for the same company?

RICH: No. That's ancient history at this point.

CHARLES: Because I'm wondering, are you actually getting to use these tools that you've been building to actually do the types of visualizations and stuff that we've been talking about?

RICH: Very much so. I moved to The Guardian some years ago. And then from there, moved to Guardian US, which has an office in New York. And it was there that I started working on Svelte. I then moved to the New York Times and I'm still working on Svelte. I've used it a number of times to build things at the New York Times and the people have built things with it too. And so, yeah, it's very much informed by the demands of building high performance interactive applications on a very tight deadline.

CHARLES: Okay, cool. So I've probably used, I mean, I'm an avid reader of both Guardian and the New York Times, so I've probably used a bunch of these visualizations. I had no idea what was driving them. I just assumed it was all D3.

RICH: There is a lot of D3. Mike Bostock, the creator of D3, he was a linchpin at the graphics department for many years. Unfortunately we didn't overlap. He left the Times before I joined the Times, but his presence is still very much felt in the department. And a lot of people who are entering the industry, they're still becoming database practitioners by learning from D3 examples. It's been a hugely influential thing in our industry.

TARAS: How long is a typical project? How long would it take to put together a visualization for an article that we typically see?

RICH: It varies wildly. The graphics desk is about 50 strong and they will turn around things within a day. Like when the Notre Dame burnt down a couple of months ago, my colleagues turned around this interactive scroll driven webGL 3D reconstruction of how the fire spreads through the cathedral in less than 24 hours, which was absolutely mind blowing. But at the same time, there are projects that will take months. I work on the investigations team at the Times. And so, I'm working with people who are investigating stories for the best part of the year or sometimes more. And I'm building graphics for those. And so that, it's two very different timescales, but you need to be able to accommodate all of those different possibilities.

CHARLES: So, what does the software development practice look like? I mean, because it sounds like some of this stuff, are you just throwing it together? I guess what I mean by that is, I guess the projects that we typically work on, three months is kind of a minimum that you would expect. So, you go into it, we need to make sure we've got good collaboration practices around source control and continuous integration and testing and all this stuff. But I mean, you're talking about compressing that entire process into a matter of hours. So what, do you just throw right out the window? What do you say? "We're just doing a live version of this."

RICH: Our collaboration processes consist of sitting near each other. And when the time calls for it, getting in the same room as each other and just hammering stuff out on the laptop together. There's no time for messing around with continuous integration and writing tests. No one writes tests in the news graphics, it's just not a thing.

CHARLES: Right. But then for those projects that stretch into like three months, I imagine there are some. Do you run into like quality concerns or things like that where you do have to take into account some of those practices? I'm just so curious because it sounds like there's actually, the difference between two hours and two months is, that's several orders of magnitude and complexity of what you're developing.

RICH: It is. Although I haven't worked on a news project yet that has involved tests. And I know that's a shocking admission to a lot of people who have a development background, but it's just not part of the culture. And I guess the main difference between the codebase for a two-hour project and a two-month project is that the two-month project will strive to have some reasonable components. And that's, I think, the main thing that I've been able to get out of working on the kinds of projects that I do is instead of just throwing code at the page until it works, we actually have a bit of time to extract out common functionality and make components that can be used in subsequent interactives. So, things like scroll driven storytelling, that's much easier for me now than it was when I first built a scroll driven storytelling component like a couple of years ago.

CHARLES: Yeah. That was actually literally my next question is how do you bridge that, given that you've got kind of this frothy experimentation, but you are being, sounds like, very deliberate about extracting those tools and extracting those common components? And how do you find the time to even do that?

RICH: Well, this is where the component driven mindset comes in really handy, I think. I think that five or 10 years ago when people thought in terms of libraries and scripts, there wasn't like that good unit of reusability that wasn't the sort of all encompassing, like a component is just the right level of atomicity or whatever the word is. It makes sense to have things that are reusable but also very easy to tweak and manipulate and adapt to your current situation. And so, I think that the advent of component oriented development is actually quite big for those of us working in this space. And it hasn't really caught on yet to a huge degree because like I say, a lot of people are still coming with this kind of D3 script based mindset because the news industry, for some interesting and historical reasons, is slightly out of step with mainstream mode development in some ways. We don't use things like Babel a lot, for example.

CHARLES: That makes sense, right? I mean, the online print is not like it's a React application or it's not like the application is all encompassing, so you really need to have a light footprint, I would imagine, because it really is a script. What you're doing is scripting in the truest sense of the word where you essentially have a whole bunch of content and then you just need to kind of --

RICH: Yeah. And the light footprint that you mentioned is key because like most new sites, we have analytics on the page and we have ads and we have comments and all of these things that involve JavaScript. And by the time our code loads, all of this other stuff is already fighting for the main thread. And so, we need to get in there as fast as we can and do our work with a minimum fuss. We don't have the capacity to be loading big frameworks and messing about on the page. So that again is one of these sort of downward pressures that kind of enforces a certain type of tool to come out of the news business.

TARAS: A lot of the tooling that's available, especially on like the really fatter, bigger frameworks, the tools that you get with those frameworks, they benefit over long term. So if you have like a long running project, the weight of the abstractions, you've experienced that benefit over time and it adds up significantly. But if you're working to ship something in a day, you want something that is just like a chisel. It does exactly what you want it to do. You want to apply it in exactly the right place and you want to get it done exactly, like you want the outcome to be precise.

RICH: That's true. And I think a lot of people who have built large React apps, for example, or large Ember apps, they sort of look at Svelte and think, "Well, maybe this isn't going to be applicable to my situation," because it has this bias towards being able to very quickly produce something. And I'm not convinced that that's true. I think that if you make something easier to get started with, then you're just making it easier. If you build something that is simple for beginners to use, then you're also building something simple for experts to use. And so, I don't necessarily see it as a tradeoff, I don't think we're trading long-term maintainability for short term production. But it is certainly a suspicion that I've encountered from people.

TARAS: This is something that we've also encountered recently. It's been kind of a brewing discussion inside a front side about the fact that it seems to be that certain problems are actually better to rewrite than they are to maintain or refactor towards an end goal. And we found this, especially as the tools that we create have gotten more precise and more refined and simplified and lighter, it is actually easier to rewrite those things five times than it is to refactor it one time to a particular place that we want it to be. And it's interesting, like I find this to be very recent, this idea is blossoming in my mind very recently. I didn't observe this in the past.

CHARLES: Do you mean in the sense that like if a tool is focused enough and a tool is simple enough, then refactoring is tantamount to a rewrite if you're talking about 200 or 300 lines of code? Is that what you mean?

TARAS: Yeah. If you're sitting down to make a change or you have something in mind, it is actually easy to say, "Let's just start from scratch and then we're going to get exactly the same place in the same amount of time." But this kind of mantra of not rewriting makes me think about that, makes me question whether that's actually something that is always the right answer.

RICH: I definitely question that conventional wisdom at all levels, as well. I started a bundler called Rollup as well as Svelte more recently. And Rollup was the second JavaScript bundler that I wrote, because the first one that I wrote wasn't quite capable of doing the things that I wanted. And it was easier to just start from scratch than to try and shift the existing user base of its predecessor over to this new way of doing things. Svelte 3 is a more or less complete rewrite. Svelte has had multiple, more or less, complete rewrite. Some of them weren't breaking changes. But Svelte itself was a rewrite of an earlier project that I'd started in 2013. And so in my career, I've benefited massively from learning from having built something. But then when the time comes and you realize that you can't change it in the ways that you need to change it, just rewrite it.

And I think that at the other end of the spectrum, the recent debate about micro frontend has largely missed this point. People think that the benefit of the micro frontend is that people don't need to talk to each other, which is absolute nonsense. I think the benefit of this way of thinking about building applications is that it optimizes for this fact of life that we all agree is inevitable, which is that at some point, you're going to have to rewrite your code. And we spend so much energy trying to optimize for the stability of a code base over the long term. And in the process, lock ourselves into architectural and technical decisions that don't necessarily make sense three or four years down the line. And I think as an industry, would be a lot better placed if we all started thinking about how to optimize for rewrites.

CHARLES: So for those of us who aren't familiar, what is the debate surrounding micro frontends? This is actually something I've heard a lot about, but I've actually never heard what micro frontends actually are.

RICH: Yeah. I mean, to be clear, I don't really have a dog in this fight because I'm not building products, but the nub of it is that typically if you're building a website that maybe has like an admin page, maybe it has a a settings page, maybe it has product pages, whatever. Traditionally, these would all be parts of a single monolithic application. The micro frontend approach is to say, "Well, this team is going to own the settings page. This team is going to own the product page." And they can use whatever technologies they want to bring that about. And the detractors sort of attack a straw man version of this, "You're going to have different styles in every page. You're going to have to load Vue on one page. You're going to have to load React on the other page. It's going to be a terrible user experience," when actually its proponents aren't suggesting that at all. They're suggesting that people from these different teams coordinate a lot more that are free to deviate from some kind of grand master architectural plan when it's not suitable for a given task. And darn right. I think it means that you have a lot more agility as an engineering organization than you would if you're building this monolithic app where someone can't say, "Oh, we should use this new tool for this thing. We should use microstates when the rest of the organization is using Google docs." It's not possible. And so, you get locked into the decisions of a previous generation.

CHARLES: Right. No, it makes sense. It's funny because my first reaction is like, "Oh my goodness, that's a potential for disaster." The klaxon's going to go off in your head, but then you think, really then the work is how do you actually manage it so it doesn't become a disaster. And if you can figure that out, then yeah, there is a lot of potential.

RICH: Yeah. People always try and solve social problems with technology. You solve social problems with social solutions.

CHARLES: Right. And you have to imagine it too, it depends on the application, right? I think Amazon, the Amazon website is developed that way where they have different teams that are responsible even down to little content boxes that are up on the toolbar. And the site doesn't really, it shows, right? Like it shows like this is kind of like slapped together, but that's not what they need. They don't need it to not look like there's slight variation with the different ways that things behave. They need to be showing for their business to work. They need to be showing the right thing at the right time. And that's the overriding concern. So having it look very beautiful and very coherent isn't necessarily a thing. Same thing in Spotify, used as another example of this. I didn't know if it was called micro frontends, but I know that they've got a similar type thing, but they are clearly the experience and having it look coherent is more important. And so, they make it work somehow. And then like you're saying, it probably involves groups of people talking to other groups of people about the priorities.

So yeah, it doesn't sound to me like just like you're going to adopt micro frontends guarantees one particular set of outcomes. It really is context dependent on what you make of it.

RICH: Totally.

TARAS: I'm curious though, so with Svelte, essentially for your reactivity engine, you have to compile to get that reactive behavior.

RICH: Yeah.

TARAS: How does that play with other tools like when you actually integrate it together? I've never worked with Svelte on a large project, so I can't imagine what it looks like at scale. I was wondering if you've seen those kind of use cases and what that ends up, if there's any kind of side effects from that.

RICH: As you say, the reactivity within a component is only in the local state within that component or to state that is patched in as a prop from a parent component. But we also have this concept called a store. And a store is just a project that represents a specific value and you import it from svelte/store. And there are three types of store that you get out of the box. A writable, a readable and a derived. And a writeable is just, var count = writable (0) and then you can update that and you can set it using methods on that store. Inside your marker, you can reference or in fact inside the script block in the component, you can reference the value of that store just by prefacing it with a dollar sign. And the compiler sees that and says, "Okay, we need to subscribe to this store as value and then assign it and apply the reactivity." And that is the primary way of having state that exists outside the component hierarchy. Now, I mentioned the writable, readable, and derived are the built in stores that you get, but you can actually implement your own stores. You just need to implement this very simple contract. And so,, it's entirely possible to use that API to wrap any state management solution you have. So you can wrap redux, you can wrap microstates, you can wrap state, you can wrap whatever it is, whatever your preferred state management solution is, you can adapt it to use with Svelte. And it's very sort of idiomatic and streamlined. Like it takes care of unsubscriptions when the component is unmounted. All of that stuff is just done for you.

CHARLES: Digging a little bit deeper into the question of integration, how difficult would it be to take wholesale components that were implemented in Svelte and kind of integrate them with some other component framework like React?

RICH: If the component is a leaf node, then it's fairly straightforward. There is a project called react-svelte which is, I say project, it's like 20 lines of code and I don't think it's [inaudible] they did for Svelte 3, which I should probably do. But that allows you to use a Svelte component in the context of React application, just using the component API the same way that you would [inaudible] or whatever. You can do that inside a React component. Or you could compile the Svelte component to a web component. And this is one of the great benefits of being a compiler is that you can target different things. You can generate a regular JavaScript class and you've got an interactive application. Or you can target a server side rendering component which will just generate some html for some given state which can then later be hydrated on the client. Or you can target a web component which you can use like any other element in the context of any framework at all. And because it's a compiler, because it's discarding all of the bits of the framework that you're not using, it's not like you're bundling an entire framework to go along with your component. And I should mention while I'm talking about being able to target different outputs, we can also, as a NativeScript project, you can target iOS and Android that same way. Where it gets a little bit more complicated is if it's not a leaf node. If you want to have a React app that contains a Svelte component that has React [inaudible], then things start to get a little bit more unwieldy, I think. It's probably technically possible, but I don't know that I would recommend it. But the point is that it is definitely possible to incrementally adopt Svelte inside an existing application, should that be what you need to do.

CHARLES: You said there's a NativeScript project, but it sounds to me like you shouldn't necessarily need NativeScript, right? If you're a compiler, you can actually target Android and you could target iOS directly instead of having NativeScript as an intermediary, right?

RICH: Yes. If, if we had the time to do the work, then yes. I think the big thing there would be getting styles to work because Svelte components have styles. And a regular style tag just to CSS and you can't just throw CSS in a native app.

CHARLES: Right. Sometimes, I feel like it'd be a lot cooler if you could.

[Laughter]

RICH: NativeScript really is doing a lot of heavy lifting. Basically what it's doing is it's providing a fake dom. And so, what the NativeScript does is it targets that dom instead of the real dom and then NativeScript turns that into the native instructions.

CHARLES: Okay. And you can do that because you're a compiler.

TARAS: Compilers has been on our radar for some time, but I'm curious like what is your process for figuring out what it should compile to? Like how do you arrive at the final compile output? Manually, have you written that code and then, "I'm going to now change this to be dynamically generated." Or like how do you figure out what the output should be?

RICH: That's pretty much it. Certainly, when the project started, it was a case of, I'm going to think like a compiler, I'm going to hand convert this declarative component code into some framework plus JavaScript. And then once that's done, sort of work backwards and figure out how a compiler would generate that code. And then the process, you do learn certain things about what the points of reusability are, which things should be abstracted out into a shared internal helper library and what things should be generated in line. The whole process is designed to produce output that is easy for a human to understand and reason about. It's not like what you would imagine compile [inaudible] to be like, it's not completely inscrutable. It's designed to be, even to that level of being well formatted, it's designed to be something that someone can look at and understand what the compiler was thinking at that moment. And there's definitely ways that we could change and improve it. There are some places where there's more duplication than we need to have. There are some places where we should be using classes instead of closures for performance and memory benefits. But these are all things that once you've got that base, having gone through that process, that you can begin to iterate on.

CHARLES: It's always curious to me about when is the proper time to move to a compiler, because when you're doing everything at runtime, there's more flexibility there. But at what point do you decide, "You know what? I know that these pathways are so well worn that I'm going to lay down pavement. And I'm going to write a compiler." What was the decision process in your mind about, "Okay, now it's time." Because I think that that's maybe not a thought that occurs to most of us. It's like, "I had to write a compiler for this." Is this something that people should do more often?

RICH: The [inaudible] of 'this should be a compiler' is one that is worth sort of having at the back of your head. I think there are a lot of opportunities not just in DUI framework space but in general, like is there some way that we can take this work that is currently happening at runtime and shift it into a step that only happens once. That obviously benefits users. And very often we find that benefits developers as well. I don't think there was a point at which I said, "Oh, this stuff that's happening at runtime should be happening at compile time." It was more, I mean, the actual origin has felt that it was a brain worm that someone else infected me with. Judgment is a very well known figure in the JavaScript world. He had been working on this exact idea but hadn't taken it to the point where he was ready to open source it. But he had shared like his findings and the general idea and I was just immediately smitten with this concept of getting rid of the framework runtime. At the time, the big conversation happening in the JavaScript community was about the fact that we're shipping too much JavaScript and it's affecting startup performance time. And so the initial thought was, "Well, maybe we can solve that problem by just not having the runtime." And so, that was the starting point with Svelte. Over time, I've come to realize that that is maybe not the main benefit. That is just one of the benefits that you get from this approach. You also get much faster update performance because you don't have to do this fairly expensive virtual dom different process. Lately, I've come to think that the biggest win from it is that you can write a lot less code. If you're a compiler, then you're not kind of hemmed in by the constraints of the language, so you can almost invent your own language. And if you can do that, then you can do the same things that you have been doing with an API in the language itself. And that's the basis of our system of reactivity, for example. We can build these apps that are smaller and by extension, less bug prone and more maintainable.

I just wanted to quickly address the point you made about flexibility. This is a theoretical downside of being a compiler. We're throwing away the constraints about the code needing to be something that runs in the browser, but we're adding a constraint, which is that the code needs to be statically analyzable. And in theory, that results in a loss of flexibility. In practice, we haven't found that to affect the things that we can build. And I think that a lot of times when people have this conversation, they're focusing on the sort of academic concepts of flexibility. But what matters is what can you build? How easy is it to build a certain thing? And so if empirically you find that you're not restricted in the things that you can build and you can build the same things much faster, then that academic notion of flexibility doesn't, to my mind, have any real value.

CHARLES: Hearing you talk reminded me of kind of a quote that I heard that always stuck with me back from early in my career. I came into programming through Perl. Perl was my first language and Perl is a very weird language. But among other things, you can actually just change the way that Perl parses code. You can write Perl that makes Perl not throw, if that makes any sense. And when asked about this feature, the guy, Larry Wall, who came up with Perl, he's like, "You program Perl, but really what you're doing is you're programming Perl with a set of semantics that you've negotiated with the compiler." And that was kind of a funny way of saying like, "You get to extend the compiler yourself." Here's like the default set of things that you can do with our compiler, but if you want to tweak it or add or modify, you can do that. And so, you can utilize the same functionality that makes it powerful in the first place. You can kind of inject that whole mode of operation into the entire workflow. Does that make sense? That's like a long way of saying, have you thought about, and is it possible to kind of extend the Svelte compiler as part of a customization or as part of the Svelte programming experience?

RICH: We have a very rudimentary version of that, which is pre-processing. There's an API that comes with Svelte called preprocess. And the idea there is that you can pass in some code and it will do some very basic, like it will extract your styles, it will extract your script and it will extract your markup. And then it will give you the opportunity to replace those things with something else. So for example, you could write some futuristic JavaScript and then compile it with Babel before it gets passed to the Svelte compiler, which uses acorn and therefore needs to be able to have managed other scripts so that it can construct an abstract syntax tree. A more extreme version of that, people can use [inaudible] to write their markup instead of html. You can use Sass and Less and things like that. Generally, I don't recommend that people do because it adds these moving parts and it makes like a lot of bug reports of people just trying to figure out how to get these different moving parts to operate together. I don't know, it means that your editor plugins can't understand what's inside your style tag all of a sudden and stuff like that. So, it definitely adds some complexity, but it is possible.

At the other end, at a slightly more extreme level, we have talked about making the cogeneration part plugable so that for example, the default renderer and the SSR renderer are just two examples of something that plugs into the compiler that says, "Here is the component, here's the abstract syntax tree, here's some metadata about which values are in scope," all of this stuff and then go away and generate some code from this. We haven't done that so far, partly because there hasn't been a great demand for it, but also because it's really complicated. As soon as you turn something into a plugin platform, you just magnify the number of connection points and the number of ways that things could go wrong by an order of magnitude. And so, we've been a little bit wary of doing that, but it is something that we've talked about primarily in the context of being able to do new and interesting things like target webGL directly or target the command line. There are renders for React that let you build command line apps using React components. And like we've talked about, maybe we should be able to do that. Native is another example. The NativeScript integration as you say, it could be replaced with the compiler doing that work directly, but for that to work presently, that would mean that all of that logic would need to sit in core. And it would be nice if that could be just another extension to the compiler. We're talking about a lot of engineering effort and there's higher priority items on our to do list at the moment. So, it's filed under one day.

CHARLES: Right. What are those high priority items?

RICH: The biggest thing I think at the moment is TypeScript integration. Surprisingly, this is probably like the number one feature request I think is that people want to be able to write Typescript inside the Svelte components and they want to be able to get TypeScript when they import the Svelte component into something else. They want to be able to get completion [inaudible] and type checking and all the rest of it. A couple of years ago, that would've been more or less than thinkable but now it's like table stakes is that you have to have first-class TypeScript support.

CHARLES: Yeah, TypeScript is as popular as Babel these days, right?

RICH: Yeah, I think so. I don't need to be sold on the benefits. I've been using TypeScript a lot myself. Svelte is written in TypeScript, but actually being able to write it inside your components is something that would involve as hacking around in the TypeScript compiler API in a way that, I don't know if anyone actually or any of us on the team actually knows how to do. So, we just need to spend some time and do that. But obviously when you've got an open source project, you need to deal with the bugs that arise and stuff first. So, it's difficult to find time to do a big project like that.

CHARLES: So, devil's advocate here is if the compiler was open for extension, couldn't a TypeScript support be just another plugin?

RICH: It could, but then you could end up with a situation where there's multiple competing TypeScript plugins and no one's sure which ones are used and they all have slightly different characteristics. I always think it's better if these things that are common feature requests that a lot of people would benefit from, if they're built into the project themselves. I go really light in the batteries included way of developing and I think this is something that we've sort of drifted away from in the frontend world over the last few years, we've drifted away from batteries included towards do it yourself.

CHARLES: Assemble the entire thing. Step one, open the box and pour the thousand Lego pieces onto the floor.

RICH: Yeah, but it's worse than that because at least, with a Lego set, you get the Lego pieces. It's like if you had the Lego manual showing you how to build something, but you were then responsible for going out and getting the Lego pieces, that's frontend development and I don't like it.

CHARLES: Right. Yeah. I don't like that either. But still, there's a lot of people advocating directly. You really ought to be doing everything completely and totally yourself.

RICH: Yes.

CHARLES: And a lot of software development shops still operate that way.

RICH: Yeah. I find that the people advocating for that position the most loudly, they tend to be the maintainers of the projects in question. The whole small modules philosophy, they exist for the benefit primarily of library authors and framework authors, not for the benefit of developers, much less users. And the fact that the people who are building libraries and frameworks tend to have the loudest megaphones means that that mindset, that philosophy is taken as a best practice for the industry as a whole. And I think it's a mistake to think that way.

TARAS: There is also, I think, a degree of a sliding scale where you start off with like as the more experience you get, because there is more experience you get closer, you get to that kind of wanting granular control and then they kind of slides down towards granular control and then slice back up to, once you've got a lot of experience, you're like, "Okay, I don't want this control anymore." And then you kind of cast that and you get into like, "I'm now responsible for tools that my team uses," and now you're back to wanting that control because you want things to be able to click together. It's kind of like a way that your interest in that might change over time depending on your experience level and your position in the organization. So yeah, there's definitely different motivating factors. Like one of the things that we've been thinking a lot about is designing tools that are composable and granular at individual module level, but combined together into a system for consumption by regular people. So like finding those primitives that will just click together when you know how to click them together. But when you're consuming them, just feel like a holistic whole, but at the same time not being monolithic. That's a lot of things to figure out and it's a lot of things to manage over time, but that's solely the kind of things we've been thinking about a lot.

RICH: I think that's what distinguishes the good projects that are going to have a long lifespan from the projects that are maybe interesting but don't have a long shelf life is whether they're designed in such a way that permits that kind of cohesion and innovation tradeoff, if you think of it as a trade off. Anyone can build the fastest thing or the smallest thing or the whatever it is thing. But building these things in a way that feels like it was designed holistically but is also flexible enough to be used with everything else that you use, that's the real design challenge.

CHARLES: It's hard to know where to draw that line. Maybe one good example of this and, these are actually two projects that I'm not particularly a fan of, but I think they do a good job of operating this way. So, I guess in that sense, it means I can even be more honest about it. I don't particularly care for Redux or like observables, but we ended up using, in one of our last React projects, we had to choose between using Redux-Saga and Redux-Observable. The Redux-Observable worked very well for us. And I think one of the reasons is because they both had to kind of exist. They had to kind of co-exist is their own projects. Like Redux exists as its own entity and Observables exist as their own kind of whole ecosystem. And so, they put a lot of thought in like what is the natural way in which these two primitives compose together? As opposed to the Saga, which I don't want to disparage the project because I think it actually is a really good project. There's a lot of really good ideas there but because it's more like just bolted on to Redux and it doesn't exist outside of the ecosystem of Redux and the ideas can't flourish outside and figure out how it interfaces with other things. Like the true primitive is still unrevealed there. And so, whereas I feel like with Redux you actually have to really, really true primitives. Now, they're not necessarily my favorite primitives, but they are very refined and very like these do exactly what they are meant to do. And so when you find how they connect together, that experience is also really good. And the primitive that arises there I think ends up being better. Is that an example of what you guys are talking about?

RICH: Maybe. [Laughs]

TARAS: No, I think so. I mean, it's distilling to the essence, the core of what you're trying to do and then be able to combine it together. I mean, that's been kind of the thing that we've been working on at the Frontside. But also within this context, it makes me think of how does a compiler fit into that? How does that work with the compiler? It's just like when you add the compiler element, it just makes it like my mind just goes poof!

[Laughter]

CHARLES: Yeah, exactly. That's why I keep coming back to like, how do you, and maybe I haven't, you just have to kind of go through the experience, but it feels like maybe there's this cycle of like you build up the framework and then once it's well understood, you throw the framework away in favor of like just wiring it straight in there with the compiler and then you iterate on that process. Is that fair to say?

RICH: Kind of, yeah. At the moment, I'm working on this project, so I referred a moment ago to being able to target webGL directly. At the moment, the approach that I'm taking to building webGL apps is to have webGL components inside Svelte in this project called SvelteGL. And we've used it a couple of times at the Times. It's not really production ready yet, but I think it has some promise. But it's also slightly inefficient, like it needs to have all of the shade of code available for whichever path you're going to take, whatever characteristics your materials have, you need to have all of the shade of code. And if we're smart about it, then the compiler could know ahead of time which bits of shade of code it needed to include. At the moment, it just doesn't have a way of figuring that out. And so that would be an example of paving those cow paths. Like if you do try and do everything within the compiler universe, it does restrict your freedom of movement. It's true. And to qualify my earlier statements about how the small modules philosophy is to the benefit of authors over developers, it has actually enabled this huge flourishing of innovation, particularly in the React world. We've got this plethora of different state management solutions and CSS and JS solutions. And while I, as a developer, probably don't want to deal with that, I just want there to be a single correct answer. It's definitely been to the advantage of the ecosystem as a whole to have all of this experimentation. Then in the wild, there are projects like Svelte they can then take advantage of. We can say, "Oh well, having observed all of this, this is the right way to solve this problem." And so, we can kind of bake in that and take advantage of the research that other people have done. And I think we have made contributions of our own but there is a lot of stuff in Svelte like the fact that data generally flows one way instead of having [inaudible] everywhere. Things like that are the results of having seen everyone make mistakes in the past and learning from them. So, there are tradeoffs all around.

TARAS: One thing on topic of data flow here and there, one thing that I've been kind of struggling to compute is the impact of that as opposed to something where you have like one directional data flow because it seems like conceptually it's really simple. You set a property like in two way balance system, like you just propagate through stuff but we don't really have a way, you don't have any way of assessing what is the true impact of that computation. Like what is the cost of that propagation where I think it's almost easier to see the cost of that computation if you have like one directional data flow because you know that essentially everything between the moment that you invoke transition to computing the next state, that is the cost of your computation where you don't have that way of computing the result in a two way balance system. Something like Ember Run Loop or mobx or zones, Vues, reactive system. All these systems make it really difficult to understand what is the real cost of setting state. And that's something that I personally find difficult because this clarity that you have about the one directional data flow and what it takes to compute the next state, it's almost like because that cost is tangible where you're thinking about like mutation of objects and tracking their change like that cost is almost immeasurable. It just seems like a blob of changes that they have to propagate. I don't know. That's just something that I've been thinking a lot because especially with the work that we'll be doing with microstates because as you're figuring out what the next state is, you know exactly what operations are performed in a process where that might not be the case with the system that tracks changes like where you'd have with zones or with Ember Run Loop, or Vue.

RICH: I would agree with that. The times that I found it to be beneficial to deviate from the top-down ideology is when you have things like form elements and you want to bind to the values of those form elements. You want to use them in some other computation. And when you do all that by having props going in and then events going out and then you intercept the event and then you set the prop, you're basically articulating what the compiler can articulate for you more effectively anyway. And so conceptually, we have two way bindings within Svelte, but mechanically everything is top down, if that makes sense.

CHARLES: Is it because you can analyze the tree of top down and basically understanding when you can cheat. This might be really over-simplistic, but if you're kind of with the event, you're collecting the water and then you have to put it way up on top of the thing and it flows down. But if you can see the entire apparatus, you can say, "Actually, I've got this water and it's going to end up here, so I'm just going to cheat and put it over right there." Is that the type of thing that you're talking about where you're effectively getting a two way binding, but you're skipping the ceremony.

RICH: It's kind of writing the exact same code that you would write if you were doing it using events. But if you're writing it yourself, then maybe you would do something in a slightly inefficient way perhaps. For example, with some kinds of bindings, you have to be careful to avoid an infinite loop. If you have an event that triggers a state change, the state change could trigger the event again and you get this infinite loop. A compiler can guard against that. It can say this is a binding that could have that problem, so we're going to just keep track of whether the state changes as a result of the binding. And so, the compiler can sort of solve all of these really hairy problems that you had faced as a developer while also giving you the benefit in terms of being able to write much less code and write code that expresses the relationship between these two things in a more semantic and declarative way without the danger.

TARAS: This is one of the reasons why I was so excited to talk to you about this stuff, Rich, because this stuff is really interesting. I mentioned that we might, so we have a little bit more time. So I just want to mention, because I think that you might find this interesting, the [inaudible], the stuff that we were talking about that I mentioned to you before. So, I want to let Charles talk about it briefly because it's interesting, because it essentially comes down to managing asynchrony as it ties to life cycle of objects. Life cycle of objects and components are something we deal with on a regular basis. So, it's been an interesting exercise and experimenting with that. Charles, do you want to give kind of a low down?

CHARLES: Sure. It's definitely something that I'm very excited about. So, Taras gets to hear like an earful pretty much every day. But the idea behind structure concurrency, I don't know if you're familiar with it. It's something that I read a fantastic -- so people have been using this for a while in the Ember community. So Alex Matchneer, who's a friend and often time guest on the podcast created a library called ember-concurrency where he brought these ideas of structure concurrency to the ember world. But it's actually very prevalent. There's C libraries and Python libraries. There's not a generic one for JavaScript yet, but the idea is just really taking the same concepts of scope that you have with variables and with components, whether they be ember components, Svelte components, React components or whatever there is, you have a tree of components or you have a of parents and children and modeling every single asynchronous process as a tree rather than what we have now, which is kind of parallel linear stacks. You call some tick happens in the event loop and you drill down and you either edit an exception or you go straight back up. The next tick of the event loop comes, you drill down to some stack and then you go back up. A promise resolves, you do that stack. And so with structure concurrency, essentially every stack can have multiple children. And so, you can fork off multiple children. But if you have an error in any of these children, it's going to propagate up the entire tree. And so, it's essentially the same idea as components except to apply to concurrent processes. And you can do some just really, really amazing things because you don't ever have to worry about some process going rogue and you don't have to worry about coordinating all these different event loops. And one of the things that I'm discovering is that I don't need like event loops. I don't really use promises anymore. Like actually, I was watching, I think it was why I was watching your talk when you're talking about Svelte 3, when you're like -- or maybe did you write a blog post about we've got to stop saying that virtual doms are fast?

RICH: Yes, I did.

CHARLES: So I think it was that one. I was reading that one and it jived with me because it's just like, why can't we just go and do the work? We've got the event, we can just do the work. And one of the things that I'm discovering is with using the construction concurrency with generators, I'm experiencing a very similar phenomenon where these stack traces, like if there's an error, the stack traces like three lines long because you're basically doing the work and you're executing all these stacks and you're pausing them with a generator. And then when an event happens, you just resume right where you left off. There's no like, we've got this event, let's push it into this event queue that's waiting behind these three event loops. And then we're draining these queues one at a time. It's like, nope, the event happens. You can just resume right where you were. You're in the middle of a function call, in the middle of like [inaudible] block. You just go without any ceremony, without any fuss. You just go straight to where you were, and the stack and the context and all the variables and everything is there preserved exactly where you left it. So, it's really like you're just taking the book right off the shelf and going right to your bookmark and continuing along. Rather than when you've got things like the run loop in ember or the zones in angular where you have all these mechanics to reconstruct the context of where you were to make sure that you don't have some event listener. An event listeners created inside of a context and making sure that that context is either reconstructed or the event listener doesn't fire. All these problems just cease to exist when you take this approach. And so, if it's pertinent to this conversation, that was a surprising result for me was that if you're using essentially code routines to manage your concurrency, you don't need event loops, you don't need buffers, you don't need any of this other stuff. You just use the JavaScript call stack. And that's enough.

RICH: I'm not going to pretend to have fully understood everything you just said but it does sound interesting. It does have something not that dissimilar to ember's run loop because if you have two state changes right next to each other, X+=1, Y+=1, you want to have a single update resulting from those. So instead of instruments in the code such that your components are updated immediately after X+=1, it waits until the end of the event loop and then it will flush all of the pending changes simultaneously. So, what you're describing sounds quite wonderful and I hope to understand that better. You have also reminded me that Alex Matchneer implemented this idea in Svelte, it's called svelte-concurrency. And when he sent it to me, I was out in the woods somewhere and I couldn't take a look at it and it went on my mental to do list and you just brought it to the top of that to do list. So yeah, we have some common ground here, I think.

CHARLES: All right.

TARAS: This is a really, really fascinating conversation. Thank you, Rich, so much for joining us.

CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at [email protected]. Thanks and see you next time.

Philippe De Ryck joins the show to talk all things security: the importance and why you should be taking active steps, how to do it in your codebase effectively, and what can happen during a breach.

Philippe De Ryck: Pragmatic Web Security

Resources:

• OWASP Top 10
• OWASP Top 10 Proactive Controls

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at [email protected]. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right.

My name is Charles Lowell, a developer here at The Frontside. Joining me, also hosting today is Taras Mankovsky. Hello, Taras.

TARAS: Hello, hello.

CHARLES: And as always, we're going to be talking about web platforms, UI platforms, and the practices that go into them. And here to talk with us today about a pillar of the platform that I certainly don't know that much about, and so, I'm actually really happy to have this guest on to talk about it is Philippe De Ryck who owns his own company called Pragmatic Web Security. I understand you do trainings and are just generally involved in the small space. So, welcome, Philippe.

PHILIPPE: Hi. Nice to meet you.

CHARLES: Wow! I almost even don't even know where to start with this subject because I'm kind of like the hippie developer mindset where it's like, "LaÖlaÖlaÖlaÖlaÖ we're in this open land and nothing's ever bad going to happen and we're just going to put code out there," and nobody would ever take advantage of any holes or anything like that. And I think that a lot of developers share that mentality and that's how we end up with major, major security breaches. And so, like I said, this is something that I'm actually very eager to learn but I almost even don't know where to start. I need training, man.

[Laughter]

PHILIPPE: Well, that's good to hear. No, you're totally right about that. If you're not into security, it seems like this fast space for a lot is happening and you don't really know how or why and what really matters to you and should I even be worried about this. And let me start by addressing the very first thing. Yes, you should be worried because maybe you're not building something that somebody cares about but you always have something that somebody wants, even the simplest of attacks always targets a valuable resource. Just to give you a very simple idea today, cryptocurrency is all the hype and you have a taker that's just aiming to misuse your users' computers to mine crypto coins because it essentially saves them a bunch on electricity cost. So, there's always something to grab. Usually, it's data or services or worse. But even in the most minimal cases, you have hardware, you have devices, you have network capacity that somebody might want to abuse. So yes, security, I would say, always matters.

CHARLES: What's the best way to get started? You said understanding that everything we do, we're holding onto resources that might be valuable, that someone might want to seize but I'm just getting started with my application. I don't know anything about security. Where do I get started on just understanding the space? And then before I even look at tools that I wantÖ

PHILIPPE: You want the honest answer for that?

[Laughter]

PHILIPPE: The honest answer is probably hire someone who has security knowledge. I don't mean this in a bad way. I've come a very long way in my career doing what I do now. And if I look at that, if you are aiming as a developer with no knowledge about security to build a secure application, it's going to be very hard. There's a lot of things you need to know, intrinsic knowledge. These are not things you can simply read a small book in a week, you know all of these security things that you'll know what to do. So, if you have no previous experience at all, I suggest to find some help.

CHARLES: Right. It's like saying, "Hey, you've never written a data layer before but you want to go out and you want to write a massively distributed system where you have all these notes talking to each other. You're not going to read the O'Reilly book 'How to Build Distributed Systems' in a week and go out and do the same thing." It's the same thing with security. You need to understand the entire context. And there's no substitute for experience.

PHILIPPE: Sorry, I actually like that comparison because in a sense, you're right, it's like these other very complex topics you don't expect to learn that in a week or a month and right a functioning data layer. But the difference is if you fail at writing that data layer, your application is probably not going to work. While if you fail at securing the application or seeing potential vulnerabilities, it's still going to work just a bit more than you anticipated. It's going to result leaking all your data to an attacker or opening all doors so that they can gain access to your server, stuff like that. So, I would say that the consequences of not getting it right are, at least in the beginning, very invisible. It's only after things happened that it's like, "Oh, crap!" And you should pay attention to that.

CHARLES: Yeah. And then you have these back doors and these leaks that are set in stone and may be very hard to change.

PHILIPPE: Yeah, absolutely. And honestly, the worst part of the breach is for a company, it might be reputation damage. But what you really should be worried about is all that personal information that's being leaked to, most cases, publicly leaked to anyone but in other cases, it's sold on [inaudible] markets and actually abused by people looking for someone else's identity or credit card information or stuff like that. And the companies usually get away with some bad press, maybe a small dip in their stock price but eventually, they'll bounce back. But it's the users that suffer from these breaches for a very, very long time.

TARAS: What do you see the kind of hot zones around concerns that companies have around security? Because I imagine it's hard to be concerned about everything, so they're probably thinking about specific things like this thing worries us. Like what kind of things do you see companies and teams need to worry about?

PHILIPPE: That's an interesting question. You have all different kinds of companies and all different levels of security awareness and what they're worrying about. I would say if you have the companies that are not very good at or don't have very much security knowledge, they're probably not to worry about things because otherwise, they would have started investing in improving their practices. If you look at the companies that are at least very aware of the landscape, I'm not saying that anybody here is perfect, but some of the companies are actually doing quite a good job. One of the most interesting challenges today is dealing with dependencies. So, all of your packages, you depend on npm, Maven, Python packages, Ruby gems, and so on. All of them form a huge attack factor in most applications today. That's definitely a problem that a lot of companies struggle with and it's very hard to find good solutions.

TARAS: GitHub recently, I saw their vulnerability alert service that I've been getting a lot of notifications from on some of the open source libraries that we use. They have a lot of dependencies. And a lot of projects have the same dependencies. So, the moment that one notification goes on, it like lights up on all of the GitHub repos that I have. So, I have been going through and like updating dependencies and all those libraries.

PHILIPPE: Yeah, that's a very good example. Absolutely. A lot of the projects we build today usually start out by installing a bunch of dependencies. Even before you've written the first line of code, you already have a massive code base that you are relying upon. And a single vulnerability in a code base might be enough, it's not always the case, but it might be enough to compromise your application. And that leaves you, as a developer, in a very hard place because you haven't written any lines of code yet. You have built a vulnerable application and that starting point can be very terrifying. So, there's a lot of reports on this. And actually, if you want some numbers, 78% of the vulnerabilities discovered in existing applications like the ones you mentioned. If GitHub alerts you like, "Hey, there's a problem in one of your dependencies," it's often even an indirect dependency, meaning that you include a framework, if you include React or Express or whatever you're building, that one of your dependencies of one of those projects actually has a vulnerability. If you look at the trees of these packages, they get quite big that it's not dozens but it's thousands of packages that we're talking about.

CHARLES: Yeah that's the other thing is how do you know how to interpret these security vulnerabilities because some of them, we get a lot of security vulnerabilities for node packages but we only use them for our development tools to build frontend. So, if we're building a React application and there's some security vulnerability in some node packages that we're using in our build tool, then that doesn't actually get deployed to the frontend. So, maybe it's not a concern but if we were actually using it to build a server, then it would be absolutely critical. And so, how do you evaluate because the same security vulnerability is not a vulnerability in one context, but might be in another or maybe I'm thinking about it wrong. You see what I mean?

PHILIPPE: Yeah, sure. I totally get what you mean. Actually, I have observed the same things. I also get these security alerts on my projects, and sometimes it's devDependency, so it seems like we don't need to care about that. You're right in the sense that you have to assess the criticality of such a report. So, they will have a rating, a severity rating saying like, "This is a minor issue," or, "This is a major issue," that should be a first indication. And then a second thing to look at is, of course, how are these things used in practice. It's not because it's a devDependency that it's not exploitable because it all depends on what is the vulnerability. If there's an intentional malicious backdoor in the library and you're building that on your build server, it might give an attacker access to your build server. So, that might not be something you actually want to do. So in that case, it does matter. Of course, if it's only stuff you run locally, you can say like, "OK, this is less important." But usually, updating or fixing these vulnerabilities also requires less effort because there's no building and deploying to production servers either. So, it's a matter of staying up-to-date with these.

And one of the things that people struggle with is handling this in a lot of different applications. You mentioned you had a lot of GitHub repos and the vulnerability starts popping up in all of them and you have to fix and update all of them. You can imagine that major companies struggle with that, as well, especially if you have quite a few different technologies. Managing all of that is insanely hard.

CHARLES: Right, because you just usually look at it and you're like, "Oh, I've got to download this." And maybe, "I haven't used it this repo for a while. I've got to clone it up, I've got to update the dependency. I've got to make sure I run all my tests locally, then run all the tests in CI and make sure I didn't break anything by upgrading. I might have fixed closed security hole but broken my functionality." And so, make sure that that is all intact and then push it out to production. Even on the small, it's like I'm looking, "OK, maybe this is going to take me 30 to 45 minutes." But if you have four or five of those things, you're looking at half your day or maybe even the whole day being gone and that's if you have the processes in place to do those automated verification. If you have a very high confidence in your deployment pipeline which I don't think a lot of places have. So, it sounds like these are complementary, like you really need in order to keep a secure application, you have to keep it up-to-date because I think what I'm hearing is you should just evaluate all the threats. You should fix it if you can. The first part of my question is, am I kidding myself when I say, "Oh, I can ignore this one because it's just local or it's just a devDependency."

PHILIPPE: The answer to that question is briefly, I would say they are less critical.

CHARLES: That's cool.

PHILIPPE: In general, the rule is update if you can. And actually some of the tools out there that monitor vulnerabilities, they will automatically create a pull request in your repo saying to upgrade to this version and then you can automatically run your tests if you have them, and you can very quickly see whether some conflicts are generated by updating that dependency - yes or no. And in most cases, if it's a minor version bump, it's going to work as expected and you can easily push out the new version without that vulnerability. So, I would say fix if you can. If it goes quickly, then definitely fix them. But I would focus on non-devDependencies first instead of devDependencies.

CHARLES: Yeah.

PHILIPPE: Second thing I wanted to add is you paint a very grim picture saying you have to spend a lot of time updating these issues and I can totally understand that happening the very first time you look into this. There's going to be some stuff in there, I can guarantee that. But if you do this regularly, the effort becomes less and less because once you have up-to-date libraries, the problem is bad but it's not like we have 50 new vulnerabilities every day, fortunately.

CHARLES: Right.

PHILIPPE: So, once you have done that, it's going to be a bit less intensive than you might anticipate at first glance. Of course, if you're using these projects, if you're reusing the same library, then you'll have to update them everywhere. That's the downside, of course.

CHARLES: It's probably a little bit dangerous to be assessing the criticality of the security threats yourself if you're not an expert, and kind of in the same way, it's dangerous to be assessing an architecture if you don't have an expertise in our architecture, I guess is the thing, because you might not understand the threat.

PHILIPPE: Yeah, that's, again, absolutely true. It again depends very much on how it's deployed and what it's used for. That's going to be one important aspect. Another thing that might be very useful is, how deep is the dependency that creates the vulnerability or has the vulnerability? Because for example, if you have your tree of dependencies, if you dependency is like five or six levels deep, the chances of malicious data are reaching that specific vulnerability, and that specific library is going to be fairly small. Because usually, libraries have a lot of features and you only use part of them in your application. So, the other one is address of the features is just sitting there and if it's never used and it's also not exploitable. So, that might play a role as well.

I saw a presentation about a month or two months ago from how Uber manages these things and they struggled with a lot of those things as well. And they eventually decided that they really care about vulnerabilities going three levels deep. And something that goes deeper is considered to be less relevant or less urgent to update because chances of exploitability are going to be very small.

CHARLES: That's actually really interesting.

TARAS: One thing that got me thinking about something that is actually happening right now. A friend of mine has a WordPress site that was hacked. But what's interesting about WordPress, I think the fact that WordPress site was hacked is not really a surprise but I think what's interesting about that is that the frequency and the sophistication of these attacks has increased. The tooling has improved also in the WordPress ecosystem. But at the same time, I think there is actually more people that are aware of the kind of exploits that could be done. There are a lot of people going after WordPress sites, but it kind of makes me think that there's probably going to be a time when the vectors of attack for web applications are going to become pretty well known as well. Because of the architecture, there are a fewer of them. But as the awareness of the actual architecture becomes more common, I think the angles of attack are going to become more interesting. Like one of the things that I was reading about a couple days ago is that there are some researchers that found a way to attract users based on a combination of JavaScript APIs that are available in the browser. So, they are actually able to fingerprint users based on the kind of things that they're using, the application for [inaudible] extensions they have installed. I think people are going to get more creative. And that's kind of scary because we've seen this happen already in WordPress and people are greedy. So, there are going to be ways. I think there's going to be more people looking at how to get into and how to exploit these vulnerabilities.

PHILIPPE: Yeah. That's actually a couple of very good examples that illustrate the underlying issue. So, this browser-based tracking of users, it's called browser fingerprinting and it's been going on for a while. Back when I did my PhD, I had colleagues working on those things and you have other people at universities doing research on this. And yes, you can use things like JavaScript APIs in the browser to identify a particular user with a very high probability. It's not perfect but it's usually enough to identify a user for ad tracking or those purposes.

By the way, these things also have a legitimate purpose. So, they are also used to keep track of a particular user to prevent things like session hijacking or detect problem logins or stuff like that, so they can also have a legitimate use case next to tracking. But they very clearly show how security will always be a cat and mouse game. Tracking used to be easy. You just set a cookie in a browser and the cookie was there next time and you knew who the user was. And then, users became a bit more savvy. You had browser extensions trying to block listings because let's be honest, they're kind of shady. So, users probably don't want that. And then the attacker started moving towards other things and getting more advanced. And you see that in other areas of security, as well. So, I consider that a good thing because as we make things harder for attackers, they will have to get more creative and it will become more difficult to exploit or to take advantage of applications. That's the good side. The bad side or the dark side of that equation is that unfortunately, the vulnerabilities are not going away. It's not because we now have these somewhat more advanced attacks using advanced features or even CView-based vulnerabilities that the old things like SQL injection and [inaudible] have disappeared in applications. That's also not true and that means that it makes it just a bit more harder for everyone on the defensive side to build more secure applications. You're going to have to know about the old stuff and you have to learn about the new stuff.

CHARLES: Again, we come back to that idea. It's all a bit overwhelming. Aside from the solution of like, "Hey, let's hire Phillippe. Let's hire some other security expert." We were actually in your training, and obviously, I don't want to divulge all the secrets or whatever. If we were to attend your training, what do you see is the most important thing for people to know?

PHILIPPE: There's no secrets there. [Chuckles] What I teach is web security. I kind of like to think I teach that in a very structured and methodical way. But in the end, there's no secrets and I don't mind talking about this here on the podcast because I honestly believe that everyone should know as much as they can about security.

What do I teach? I can talk about specifics but I can also talk about generic things. One of the general takeaways is that one of the best things in my opinion that a developer can do is realize when they don't know something and actually admit that they don't know something, instead of just doing something. Maybe having like a brief thought like, "Hmm, is this secure? Well, it's probably good. I'm going to deploy it anyway. We'll see what happens." That is not the right way of doing things. If you do something and you recognize like, "Hey, this might be security sensitive. We're dealing with customer information here. We're dealing with healthcare information. We might want to look at what plays a role here," and then you can go ask someone who does. You probably have a colleague with a bit more security knowledge, so you can ask him like, "Hey Jim, or whatever your name is, do you think that this is OK or should we do something special here?" Very much like you are doing, asking me questions right here. That's one important takeaway that I hope everyone leaves with after a training class because not knowing something and realizing that you don't know it allows you to find someone who actually does. That still leaves us with that point which you wanted to sidestep.

CHARLES: [Chuckles]

PHILIPPE: A second thing is to realize that security is not a target. It's not something you're going to hit. It's not a holy goal that after working really hard for two years, you're going to hit this security milestone and you're done. It's always going to be a cat and mouse game. It's always going to be a moving target but that's OK. That's how things are. And that's the same with all other things in the world essentially. It's an evolving topic and you'll need to be ready to evolve with that as well.

TARAS: One of the challenges that I see into quite often in teams is that at the individual level, people really try to do their best, maybe the best of their abilities. But it's often, when it comes to being part of a group, it's often like they do best within the kind of cultural environment that exists. I'm curious if you've seen good or kind of environments or cultures for engineering teams that are conducive to good security. Are there kind of systems or processes the companies put in place that you've seen to be very effective in preventing problems? Have you encountered anything like this?

PHILIPPE: Ideally, you have developers that are very well educated about security but honestly, it's going to be insanely hard to find these people because a developer not only has to be educated about security, they also need to know about UI design and JavaScript frameworks and other frameworks and all of these things. And it's virtually impossible to find someone up-to-date on all of these things. So, what most companies do today that seems to work quite well, even though it's very hard to judge whether it's working or not, is they work with security champions. So, you typically have a dev team and within a dev team, you would have a security champion, one or two or five, depends on how large your teams are, of course, that is knowledgeable about security. So, that developer has some knowledge. He's not an expert but he knows about common attacks and common dangers and how to potentially address them in the application. So, having that person embedded in the team allows the team to be security aware because when you have a team meeting like, "Hey, how are we going to solve this particular problem?" That person will be able to inject security knowledge like, "Hey, that seems like a good idea but if we're using SQL in the backend, we need to ensure that we don't suffer from SQL injection." Or if you're using a NoSQL database, it's going to be NoSQL injection and so on. And that already elevates the level of security in the team.

And then, of course, security champions themselves are not going to be security experts. They're mainly developers just with a security focus. So, they should be able to escalate problems up to people with more knowledge, like a security team which can be a small security team within their organization that people can easily reach out to, to ask like, "Hey, we're doing something here and I know that this is security relevant and I'm not entirely sure what's happening here. So, can we get a review of this part of your application?" Or, "Can you guys sit on the meeting to see what's going on and what's happening there?" And I think that structure also makes sense. It's still going to be hard to build secure applications because there's still a lot of things to address, but at least, your teams get some awareness. And then of course, you can help your security champions to become better and they will get better over time. You can augment them with the security architects. You can train your security champions separately with more in-depth knowledge and so on. And that veteran or that setup seems to work quite well in many large organizations today.

CHARLES: Yeah. I like that. It gets me to thinking, so having the having the security champions, having people who have this as part of, not their specialization, but at least part of their focus, being in the room, being part of the conversation because we try and do that and provide that service when it comes to UI but we also have a bunch of processes that kind of automate the awareness of quality. So, the classic one is your CI pipeline, your deployment pipeline. So, you're automating your advancement to production. You're automating your QA. It's still no substitute for having someone who's thinking about how to have that quality outcome but you still have some way of verifying that the outcome is quality. Are there tools out there that you can do to kind of keep your project on the security Rails. I'm thinking something that we we've done recently is having preview apps, so that we get a tight feedback loop of being able to deploy a preview version of your application that's on a branch but it's talking to a real backend. There's a lot of more software and services that are supporting this and it's kind of become an integral part of our workflow. So, testing automated deployment preview apps, there's this kind of suite of tools to make sure that the feedback loops are tight and that the quality is verified even though you have people, you also have people guiding that quality. It's just making sure that the standards are met. Is there a similar set of tools and processes in the security space so that we've got these champions out there, they're being part of the conversations. They're making suggestions but they can't be everywhere at once. And is there a way to make sure that the kind of the ways that they're guiding the application, just verifying that the application is going in that direction? Or an alarm bell has sounded. We mentioned one which is the automated pull request with the, "Hey, you got this dependency and there was a pull request." Are there more things like that, I guess, is what I'm saying.

PHILIPPE: Yes, there are. But I would dare to say not enough. So yes, you have some security tools you can integrate in your pipeline that do some automated scanning and they tried to find certain issues and alert you of those issues. So, these things do exist but they have their limitations. A tool can scan an application. Some of the findings are going to be easy and fairly trivial, but it's good to have the check in place nonetheless. But some of the more advanced issues are very likely to be undetectable by those automated tools because they require a large amount of skill and expertise to actually craft and exploit to abuse that particular feature in an application. So, we do have some limitations but I like discretion because I do believe that we need to leverage these mechanisms to ensure that we can improve the security quality of our applications. A very simple thing you can do is you can run an automated dependency check when you build the application and you can use that to decide to halt deployment when it's a severe vulnerability or go ahead anyway when you consider this to be acceptable because if you automate all of those things, things can go wrong as well. We can talk about that in a second.

So yeah, these things can be done. But what I strongly encourage people to do to ensure that they can kind of improve the code quality is to flag certain known bad code patterns. So if you're building an Angular or a React application, if you're using functions that output go directly into the template, that's going to be very dangerous. So, we know these functions in Angular, they're called bypassSecurityTrustHtml, bypass security should be kind of a trigger and this kind of security irrelevant. And in React, that property is called Dangerously Set innerHTML, also indicating like a 'developer watch out what you're doing'. So, what you could do is you could set up code scanning tools that actually flag these things whenever they appear in application because sometimes people make mistakes. You hire an intern and they don't really know the impact of using that property and they use it anyway which would cause cross-site scripting vulnerability. If you're code scanning to flag these things ensures that it doesn't get pushed to production unless it's a benign case which is actually approved to be in there, then you can definitely stop some of these attacks coming on for sure or some of these vulnerabilities happening.

TARAS: I think the hardest thing to understand is when someone doesn't understand what they're doing that what they will create is so cryptic that I think any tool that tries to figure out what it is that person is doing I think will have a really hard time. The person making the thing doesn't understand what they're doing, then the system is not going to understand what they're doing which makes me think that one of the things that we think about a lot at Frontside is this idea of trying to understand the system from the outside as kind of looking at a system as a black box and wonder what kind of tools are available specifically for inspecting the application from the outside, like as if somehow understanding what the application is doing based on what's actually going on inside of the runtime and then notifying someone that there could be something off in the application, but through exercising the [inaudible] things like, for example, memory leaks is not something you can catch unless you have a test suite that has like a thousand tests and then you will see over time that your application is actually leaking memory. But if you run individual tests, you'll never see that. I wonder if there's anything like that for security where at runtime, there's actually a way to understand that there might be some kind of a pattern that's incorrect in the application.

PHILIPPE: If only, if only. It depends on who you ask. There is such a concept that's called Dynamic Application Security Testing. Essentially, what you do there is you run the application, you feed it all kinds of inputs, and you monitor when something bad happens. And that means that you have detected vulnerability. So, these things do exist. But unfortunately, their efficiency is not always that good. It very much depends on what kind of security problems you're trying to detect. And they can, for example, detect some instances of things like cross-site scripting or SQL injection or things like that. But there will always be limitations. I've seen tools like that being run as an application where you actually know there's a vulnerability because it has been exploited. There is a manual written exploits and the tool still doesn't find any vulnerabilities which is not surprising, because these things are really hard to make an abstraction of to be able to find that in an automated way with a tool. If you would have such a tool that would be, I think, that [inaudible] would be a lot better. I think there's a lot of funders working on that. But at the moment, those tools are not going to be our savior to build more secure applications.

CHARLES: Yes. I mean, it's kind of like linting, right? Or you can make tests. We've been through this kind of all the features or the aspects that we want our application to have, whether it be accessibility. There's certainly a very comprehensive suite of lint level checks that you can run to make sure that your application is accessible. You can run a suite of three thousand things and if it triggers any of these things, then yes, your application won't be accessible but it's not a substitute for thinking through the accessibility architecture. The same thing goes with code linting. You're not going to solve bugs with a linter that makes sure that it's formatted and that you're declaring your variables right and that you're not shadowing things. But you can definitely eliminate a whole classes of things that might be put in there just for maybe even you know what you're doing and you're just forgetful.

PHILIPPE: Yes, these rules exist, as well. They're not extensive but there are linting rules for Angular used for security, for example. But the problem in linting is that they are very useful to find potential instances of security relevant features or security relevant functionality. But the linting rule alone cannot decide whether something is OK or not. Just to give you a very simple example, if you use the bypassSecurityTrustHtml function, if you give that function a static snippet of HTML, that's going to be fine unless you write your own attack essentially. But if you feed that function user inputs, you're going to be in a lot of trouble. And making that distinction with a linter is going to be difficult unless you have a static string in the arguments. But if once you start having that from variables to dynamically decide to have a different code path, then that's going to be very, very difficult to decide automatically. So, yes, you can use that to find the places in the application where you should be looking for, in this example, a cross-site scripting in Angular but the linting alone is not going to give you an answer whether this is OK or not. That still requires knowledge of how Angular handles this things, what happens, and how you can do things safely.

TARAS: Sounds like we keep going back to nothing beats having knowledgeable developers.

PHILIPPE: Yes. Unfortunately, that is true. However, with that said, I want to highlight that frameworks like Angular, well mainly Angular, make things a lot better for developers because yes, you still need knowledgeable developers but the ways to introduce a cross-site scripting vulnerability in an Angular application are actually very, very limited. It's not going to be one, but there's going to be maybe three or four things you need to be aware of, and then you should be set. While if you would have done the same for PHP, it's going to be 50,000 things you need to be aware of that are potentially dangerous. So, yes, frameworks and libraries and all of these abstractions make it a lot better and I really like that. That's why I always refer to abstract things away in a library so that you actually have the ability to look for this dangerous code patterns using linting rules in your code base and that you can, at least, inspect the go to see whether it's OK or not, even though you might not be able to make an automatic decision. You, at least, know where to look and what to approve or how to change in the code base.

TARAS: I think that's one of the things that oftentimes is not taken into account that the frameworks are different. And I think of big differences in how much -- like right now, the most popular framework, I think, React. But it's such a thin layer, it's such a small part of the framework that you can hardly call it a framework. But it is something that companies rely on. But then when you consider how much of that code that you need to write, to make React into a complete framework for your company, the amount of code that your team has to write versus the amount of code that your team has to write when you use something like Angular or Ember, there's definitely a lot less parts of the framework that you need to write or a lot less parts of the framework you need to choose from what's available in the ecosystem. Like in Angler and Ember, and I'm not sure what the story is with the view, but the pieces, they come from kind of a trusted source and they've been kind of battle tested against a lot of applications. But I don't think that enters into consideration when companies are choosing between Angular or whatever that might be because they're thinking like what is going to be easiest for us. What is going be [inaudible] for developers? They're not thinking about how much of the framework are we going to need to put together to make this work.

CHARLES: I can say it sounds, Taras, like almost what you're saying is by using the frameworks that have been battle tested, you actually get to avail yourself of code that actually has security champions kind of baked into it, right? Is that what you were saying? You keep coming back to 'you need developers who are knowledgeable about security', and if you're using kind of a larger framework that covers more use cases, you're going to get that. Do you think that that is generally true, Philippe?

PHILIPPE: Yeah. I think it is and that's why I mentioned that I liked Angular before because Angular actually does offer a full framework. And because they do that, they made a lot of choices for developers and some of these choices have a very, very big and positive impact on security. On the other hand, if you make those decisions, you become an opinionated framework and some people don't like that. They actually want the freedom to follow their own paths and then a less full featured framework like React might be an easier way to go.

CHARLES: But I think what happens is folks don't enter into that decision with their eyes open to the fact that they then now need to be their own security champion because they just don't even see it. We said the most dangerous thing is the things that you don't know.

PHILIPPE: Yeah, absolutely. And I totally agree. That's something that's at least a couple of years and probably still today, many companies moving into this space struggle like, "Which framework do we choose and why do we choose one or the other and which one will still be there in three years because we don't want to switch to another thing in three years," which is risky to our developers. I like that you said that Angular has this security champion knowledge built in because in Angular 2 and every version behind it, but the new version of Angular essentially, they spent a lot of time on security and they learned from their mistakes in the first version because there were some and they took that and they built a more robust framework with security built in by design or by out-of-the-box. Angular offers, for example, very strong protection against cross-site scripting. It's just there, it's always on and unless you actively sidestep it, it's going to protect you. And that's one of the things I really like about Angular and how they did that.

CHARLES: Yeah, that's one of the things that I really like too because I remember there was a blog post back, this is probably, I don't know, almost 10 years ago now, maybe seven or eight years, where someone was comparing why they were more interested in using, their servers were implemented in Ruby and why it was better to use Rails than just Sinatra which is just a very, very, very lightweight HTTP framework. And one of the things that he was pointing to was this new vulnerability was discovered and if you were using Rails, the middle way where the middle square stack is managed by the framework, you just upgrade a minor version of Rails. And now, by default, there's this middleware that prevents this entire class of attack.

PHILIPPE: Was that a cross-site request forgery?

CHARLES: I think it might have been.

PHILIPPE: I think Rails was one of the first to offer built in automatically on support for that. So yeah, that was a very good early example of how that can work really well.

CHARLES: And the advantage from the developers' standpoint, because the contrast that, if you'd been writing your application in Sinatra which is this is very, very low level based right on top of rack and you're managing the middleware stack yourself and there are no opinions, then not only do you have to like fix this security vulnerability, you have to understand it. You have to get to do a lot of research to really come up with what's going on, how is this going to affect my application and then I can deploy a fix. And that's like a huge amount of time, whereas you have the freedom to not even understand the attack. I mean, it's always better to understand but you can defer that understanding invariably knowing that you're kind of invulnerable to it. And I think for people who enjoy kind of pretending, not pretending, but that the security world doesn't exist and say, "Hey, I want to focus and specialize on these other areas and attain deep knowledge there." It's very reassuring to know that if a defense for a novel attack comes out, I can avail myself of it just by bumping a version number.

PHILIPPE: Yeah, absolutely. If you have everything in place to actually upgrade to that version that fixes those, that's a preferable solution. Towards the future, I believe it's going to be crucial to ensure that we can actually keep things up-to-date because everything that's being built today is going to require continuous updates for the lifetime of the application. I definitely hope that the frameworks get better and more secure and start following these patterns of naming the potentially insecure functions with something that indicates that they are insecure. I think that's definitely a good way forward.

CHARLES: Yeah. Can I ask one more question? Because this is something that is always something that I wonder about whenever you talk about any aspect of a system. And part of it is folks will not appreciate good architecture until they've experienced some sort of pain associated with not having that architecture in place. Their project fails because they couldn't implement a set of features without it taking months and years and they just ran out of runway, ran out of deadline. Those types of people who've been on those projects appreciate having a nimble system internally, good tooling. Folks who have experienced good tooling understand how much time they could save, and so, have a very low tolerance for bad tooling. A tool takes too long or is misbehaved or is not well put together, they just can't stand because they know how much time they're losing with security. Is there a way to get people to care about it without having some sort of breach, without having gotten smacked in the face? When you do your trainings, is it generally, "Hey, someone has experienced a breach here, and so they want to bring you in." Or is there some way to get people raise awareness of the problems they don't have to experience that pain but can just experience only the benefit?

PHILIPPE: That's, again, a very good question and that's also a very good illustration of why security is so hard. Because if you get everything right, nothing happens.

[Laughter]

PHILIPPE: Or it might be if nothing happens, that nobody cares enough to actually try something against your application. So, there's no positive confirmation if you've done a good job. You can keep putting things off but eventually, there's going to be vulnerability and it's a matter of how you respond to it. We recently had a cross-site scripting in Google's homepage, one of the most visited pages on the web. And somebody figured out that there were some weird browser thing that could be abused and that resulted in a vulnerability on, let's say, such a simple page. So, even there, things can go wrong. So, what would be a good way to draw with some awareness about this is I would recommend following some simple new resources or some Twitter feeds. I have some security relevant articles there but plenty of other people in the industry have as well. And when you read such an article about security incidents, just think about whether this could happen to you or not. And that should probably scare the shit out of you. Simple examples like the Equifax breach, one of the biggest, most impactful breaches of the past few years happened because of an Apache library that was not updated. I think the Apache library, they had a known vulnerability in there. We knew about it. We had a patch, yet it took too long to install that patch and the attackers abused that vulnerability. This is something that probably can happen to each and every one of us because the attacks started, I think, 72 hours after the vulnerability and the patch had been published. So, ask yourself, "Would I have updated my servers in three days after I got that vulnerability report on GitHub?" Yes or no. And if the answer is no, then the same thing can happen to you.

Other cases: Magecart is a very big problem, people injecting credit card skimming malware in the JavaScript library. Are you including third party JavaScript libraries? If yes, then chances are that this can happen to you. And there's nothing preventing someone from exploiting that. It's probably just because you got lucky that nobody tried to do that. And the same thing you see now with all these attacks npm packages where people actively try to get you to install a malicious package as one of your dependencies. And again, everybody can fall victim to these things. So, if you read the articles with that mindset, I probably guess that your security awareness will grow rapidly and you will start caring about that very fast.

CHARLES: Yeah.

TARAS: Lots to think about.

CHARLES: Yeah, there's lots to think about because the next thing that occurs to me is how do you even know if you've been targeted. Because a good attacker is not even going to let you know.

PHILIPPE: Yeah.

CHARLES: It's just better to siphon off your blood, like you said, than to kill the -- you want to be a vampire bat and come to the same cow every night and just take a little bit of blood rather than the lion that kills the cow and then the cow's gone.

PHILIPPE: I would say constant monitoring is going to be crucial and you need that data for all kinds of different purposes. You need to monitor everything that happens, first of all, for a post-mortem analysis. If something happens, you want to be able to see how bad it was. This user apparently got a full admin access and if you have decent monitoring, you will be able to retrace his steps to see what they did or did not get. So, that is one very good use case. A second use case is you can use that data to detect attacks. Usually when the attacks are noisy, it's an automated scanning tool but it might be an attacker trying to do things. Again, that may be something very useful for you to act on to see if there is a problem to prevent that user from connecting, or so on. And then, another very good use case of these things is actually inspecting the logs manually as an ops engineer or whatever, who is responsible for doing that, because that might again yield new insights. I've been talking to someone who said that they discovered an abuse of one of their APIs just by looking at the logs manually and detecting a strange pattern and looking and digging deeper into it. And the automated monitoring tools that they had installed that trigger on certain events like a mass amount of requests to the authentication and stuff like that, they did not catch this particular abuse. So, I would say monitoring there is absolutely crucial, for sure.

TARAS: So, the takeaway is higher attentive knowledgeable developers who will learn about security.

PHILIPPE: I would say the takeaway is security knowledge is essential for every developer. So, I encourage every developer to at least have a little bit of interest in security. I'm not saying that everyone should be a security expert. We should at least know about that the most common vulnerabilities in web applications, what they mean, what they might result in, and what to be on the lookout for. So yes, I think that's one of the crucial things to start with. And then within an organization, you should have someone to fall back on in case that there are security relevant things that you actually can talk to someone who does see a bigger picture or maybe the full security picture to decide whether these things are a problem or not.

I think we're closing or nearing the end here, but one of the things we haven't talked about is how to actually get started in security. What if you are interested in security after hearing this podcast and you want to get started? I want to give you just a few pointers so that you actually know where to look.

One of the first things to look at is OWASP. And OWASP is the Open Web Application Security Project. It's essentially a nonprofit that has the mission to improve the security posture or knowledge of developers, and they have a lot of resources on various different topics. They have a lot of tools available and things like that. What you want to start with as a developer is the OWASP Top 10, which is a list of the 10 most common vulnerabilities that exist in applications, just to open your eyes like these things exist in applications today and are definitely a problem. And then, there's a complementary Top 10 called the Proactive Controls and that's about how you, as a developer, can actually prevent these things. So, what should we know about implementing security, which guidelines should we follow. And these two documents are a very good place to start. And then there is a huge community that's actually mostly very eager to help people figure out the right way of doing things and solving these problems we have in our ecosystems.

TARAS: Awesome. That's great. Thank you very much.

CHARLES: Yeah. I'll take that in. That is really, really helpful. Well, thank you very much, Philippe, for coming on and talking about security. I actually feel a lot better rather than usually I'm thinking about securities kind of stresses me out. [Laughs]

PHILIPPE: You can bury the problem but it's going to return in the future anyway, so you might as well get onboard and start learning. It's not that scary if you actually -- it's a lot of fun. So, you should learn about security.

CHARLES: Well, I am looking forward to diving in. If anyone wants to get in touch with you, how would they do that on Twitter or Email?

PHILIPPE: Yeah, sure. I'm on Twitter. I'm always happy to chat about security. You can reach me by Email as well. I'm very easy to reach. And I'll be happy to help out people with questions. Sure.

CHARLES: All right. Thank you so much, Philippe.

Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old Email at [email protected]. Thanks and see you next time.

In this internal Frontside Podcast episode, Charles, Taras, and Jeffrey analyze the NativeScript Mobile Platform.

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at [email protected]. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles, a developer here at Frontside. With me today are Taras and Jeffrey.

TARAS: Hello everyone.

CHARLES: Today, we're going to be talking about NativeScript, in particular, and evaluating technologies and frameworks, kind of at the meta level. So, I'm kind of excited about it because we've been pretty heavily involved with NativeScript for the past three months or so. And so, we've gotten to look at it both from beginners' eyes being kind of totally fresh to the platform, but then actually having to start to pump up against some of the edge cases which is what always ends up happening when you actually use a framework for real. Let's get started.

TARAS: All right. I think there's a lot of things that we could talk about because when we would start looking at NativeScript, the length that we were looking at NativeScript through this is that this platform that our client is going to be using for doing development of large applications. So, what does NativeScript need to have to be able to support potentially hundreds of developers building apps? We started looking at it and one things that made us consider NativeScript early on was it kind of provides a platform that allows you to encode in JavaScript and run it on mobile. And we saw this kind of emergence of Angular and Vue.js running on top of NativeScript. So, those things together is kind of exciting.

CHARLES: There was also an implementation in progress of React and there were a couple of spikes of Ember also running on top of NativeScript. So, my first impression was initially very favorable. The onboarding experience is actually pretty nice because it was JavaScript and the application was interpreted, there's the ability to completely and totally dynamically change the application at runtime. So, they have essentially an application called the NativeScript Playground which lets you flash a QR code at it and then it will go in to the URL associated with that QR code and it will download all of the assets for a NativeScript application running at that URL. So, all the JavaScript, all the templates, all the whatever, it'll pull it down, it will actually start running like within that app. So, the Playground app then becomes your actual app that you want to use. There's no App Store, no TestFlight, no Google Play. There's no gatekeeping to delivering your application into a running app. And I thought that was really, really cool and really, really compelling.

TARAS: We should clarify that this is specifically for preview purposes because if you're going to be shipping the application to production, you still need to go through all those things before...

CHARLES: Yes.

TARAS: But the onboarding process, you could just install the preview app and then you can point a QR code and it will open that app, whether it's in Angular or in Vue, that app will open up in the preview app and you have a native app that you could play around with.

CHARLES: Right.

JEFFREY: And that's key both for the engineers who are playing around with this and building this and also really key for the non-engineers who are part of the team to be able to really easily spin up and see what the engineers on the team are working on.

CHARLES: That's exactly why we thought, "Hey, we want to be able to use this mechanism for preview apps." In the same way on the server side, you have preview apps associated with a pull request. When we saw this, what we immediately wanted to do was have a bot post a comment onto a pull request with a QR code, so that anybody could just, boom, test out this app on their phone.

TARAS: We ultimately ended up setting that up but not quite that way because the original idea of being able to have something like danger bot post the QR code to the comments, you can kind of point out with your phone and open the preview app, that didn't actually pan out. Charles tried to implement that. What happened there?

CHARLES: What it actually turned out was that the preview functionality was dependent on a central server, a central NativeScript server. So rather than kind of statically bundling the assets and just saying 'these assets are this URL and just pull them in and bootstrap your NativeScript application that way', it required a lot of extra stuff. So, it required you to be running a Webpack Dev Server that was building your assets and then basically registering and doing some port forwarding with that dev server to a central NativeScript service that was provided by the company that underpins NativeScript. And that connection needed to be hot and live the whole time for that to work.

So, while it was really cool that you could get the QR codes up and running, unfortunately that functionality could not be decoupled from the hot update and the central service. Those central services were kind of hard coded into the tools.

TARAS: Yeah. So we eventually ended up implementing the preview apps that we wanted but we ended up using Appetize.io to essentially -- the process there is you build the app, you upload the app to Appetize and then danger bot embeds a link to a URL where you can open that app and it will essentially stream like it's running somewhere in a simulator for iOS, an emulator for Android and it will stream a video of that and you can interact with it, kind of like a VNC setup.

CHARLES: Yeah.

TARAS: And that actually accomplished the goal. It's just we weren't able to do the way that we thought we were hoping to do it straight off with the preview app mechanism.

CHARLES: It accomplished the goal. And Appetize is an incredible service that lets you preview the apps on pretty much any type of Android device, any type of iOS device, right there inside of a pull request. But what it didn't allow us to do was pop up your actual device, your actual phone and scan a QR code off of the pull request and pull down the assets. That would have been amazing. But it doesn't always work out that way. And I don't know if that would work long term anyhow because you can't pull down native libraries over the wire and funk them in. That's a big, big no-no. So, the process does have limitations. But nevertheless, that part was really cool.

TARAS: Yeah. That was kind of the entry point, the onboarding. And then I think one of the things that was kind of, I remember at the time when we were talking about the NativeScript architecture because we were starting to understand more about how it works. The idea itself is really kind of amazing actually because you have this V8 where you can run your JavaScript code and then they're kind of wired together on iOS and Android. They're wired to the native implementation. So when you're interacting with it, I think the thing that's really great about NativeScript is that the runtime environment for JavaScript essentially gives you API access. In JavaScript, you could say, "I want to create a Java view," and there will be a Java view that's rendered in the actual native device. You're using the same -- the APIs that you find on the Android docs or iOS docs, all of those APIs are available to you as JavaScript. So, you [crosstalk] as JavaScript. And it's seamless, right?

CHARLES: Yeah, and it makes it very, very handy. The language is different but the APIs are exactly the same. There is an attempt to make cross-platform components and cross-platform classes that serve the needs on both platforms and then delegate to the platform on which you happen to be running. But those are not mandatory, and the low level APIs are always available to you. An example of this is in iOS, kind of the core foundational object is NSObject. All the controllers, the views, the things, all of them are descended from this object. I can go from object and I can go in from JavaScript and I can just say {let object = new NSObject} and boom! I've got a reference to the actual object and I can pass it around to any other iOS API.

That is really, really powerful that there's nothing off limits. There's nothing at an arm's distance. There's really not much you can't do because all of those things are available to you. There's nothing that's off limits. That means that they can build cross-platform components on top of those APIs. Whereas a sort of system like React Native which does have cross-platform components, that's kind of where the base layer is but you can't crack open the hatch and go down the next level and start mucking around, unless you want to actually start meddling with the React Native source code or recompiling Swift in Java code.

TARAS: For me, I think this architecture is probably my favorite part of NativeScript.

JEFFREY: Mine too.

CHARLES: Yeah, me too.

TARAS: I really like this part. I kind of hope that everything else is as clever as that was.

CHARLES: Because among other things, it allowed us to write a Bluetooth. We were able to implement Bluetooth using nothing but JavaScript. We didn't actually have to go down and do any Swift and do any asynchronous message passing between the iOS libraries and the JavaScript libraries. It's like, "No." We've just got a very simple cross-platform interface that instantiates an implementation for Android and an implementation for iOS, but both of them are like JavaScript. And so, it really is you're doing native development but it's JavaScript all the way down.

TARAS: Yeah. And when you're writing plugins, your plugin is actually JavaScript plugin that is assuming iOS APIs and Android APIs.

CHARLES: Yeah. And if you have to have a native plugin like a CocoaPod or an Android Package, you just install it and you can instantiate it from JavaScript. There's no fuss, no muss, no ceremony. It's just like, "Hey, I want to use the..." what was the one we like to use? The Material-UI floating button which is a CocoaPod. You download it, you link it into your application, and then you just instantiate it from JavaScript.

TARAS: That was really cool. The challenging part was that a lot of that kind of awesomeness, like everything around it wasn't quite as polished. And so, one of the big things is that like around tooling, because one of the things about having grown up in a way like in the Ember community, in a sense, we have a certain expectation of what the level of polish from tooling that we would expect. And it's kind of supported in the way like when you look at how React or React Native tooling is, even Angular tooling, it's very polished. You kind of expect to see what you need to see when you're looking at a CLI input and you don't see anything else. That level of polish. I think part of the changes that they're going through, maybe that's part of the reason but that same level of polish isn't available around the tooling.

CHARLES: There are these fantastic qualities about the platform and it is amazing. We were using Angular and a lot of people are using Vue and things like that and that actually is pretty incredible. And there is nice tooling, there is command line stuff, but we started to run into issues where, for example, it was very clear that we were pretty much, as far as I could tell, one of the very, very few people running a NativeScript project on CircleCI or in a CI environment at all. It had capability for testing, both for acceptance testing and for unit testing, but it required changes to the core framework and the core tools in order to get those tests to work in a CI environment.

JEFFREY: Before we kind of get into the testing story there, some of the issues were around determinism of reliably reproducing your whole NativeScript environment and stack every time because that's such a key feature of doing it. And on a CI server, it's like, "Hey, we need this to load in the same exact packages every time." And so, we ran into challenges there.

TARAS: I think we spent almost two days. There's example projects in different combinations. One thing that was off was that there's a pattern that is applied in a lot of the plugins in NativeScript ecosystem is installing things. So, you run npm install and npm install will generate some files. And so, when we're trying to move it over to a CI, there were files, like there's hooks, like TypeScript hooks that were excluded that you can ignore, but they were necessary to compile the TypeScript. And so, what was happening is when we're running these at CI, the application, we would build the app but the app would crash the moment that you start it. And the reason for that was that the JavaScript files that were transpiled from TypeScript to JavaScript, those JavaScript files were actually never included because they were never transpiled in CI because the hooks directory, like we weren't preserving it between our tasks and so...

CHARLES: Right. We weren't caching. This was an artifact of the install. And so, we were caching the install, so essentially the yarn.lock was not changing. But the directory was not getting generated unless the cache key changed.

TARAS: And we spent spent quite a lot of time...

CHARLES: Two or three days out.

TARAS: Yeah.

CHARLES: What that said is, "Oh, nobody's really running this in CI." Nobody's actually building an app from scratch every time.

TARAS: There are people in NativeScript team that actually does a great job of documenting. They did have example projects that exist but sometimes that example project doesn't fit like a perfect combination of what you're looking for. There was an example project that was showing how to run on CI but it didn't use TypeScript. And so, that's where we lost a lot of time.

CHARLES: Right.

JEFFREY: So, let's talk about testing since that's kind of the core, the most important part of why you even want continuous integration capabilities to begin with. What did we run into there? What did it look like?

TARAS: Well, I think it's safe to say that we were really on a bleeding edge of testing capabilities in NativeScript ecosystem with Angular, at least. But I think it was still an interesting project. We were using the latest builds. And I have to say I think this is one of those things that's going to be kind of consistent through this, is like the people in NativeScript team are amazing. They're so easy to work with. They're so accommodating. When we ask for stuff, they're on it. But it was a lot of things we're trying to figure out like how do we run unit tests, what can we do. Ideally, we wanted to run, first and foremost, we started with how do we run functional testing. So we spent quite a lot of time trying to get Appium set up. I spent a good two to three weeks on that and it was not productively spent time.

CHARLES: I think ultimately, we had to pull back from it. And there were a number of reasons. Part of that is there are multiple paradigms for how you can build your NativeScript application. So as we speak, there's a move towards using Webpack to build all of your JavaScript in your style sheet assets because it's very much like a React Native application. You've got style sheets, you've got JavaScript assets, that some of them might be in TypeScript, some of them you might be using Babel, and you need to actually transpile them down to include them in a way that your underlying JavaScript runtime is going to be able to understand. But that wasn't always so. They have their own build system and packaging system, they kind of used the TypeScript compiler ad-hoc, if you were using TypeScript, which we were. And so, this was kind of this orthogonal complexity, I guess, where you have your unit testing and it has to play nice with this one package or Webpack.

There were multiple ways to package your app. And so, we ran into problems where, like TypeScript kept coming up as a problem and the way in which we were bundling our assets. So, in order to get TypeScript to work, we kind of had to get Webpack running. But the problem is it felt like three quarters of the tooling wasn't Webpack compatible yet. And so, it meant that other pieces of the build were breaking because of this. And so, we had to be on the bleeding edge of several different aspects of the runtime. And the problem is when you're on the bleeding edge, that can break other stuff.

TARAS: But there's complexity in running on native platforms that I think a lot of this complexity is kind of leaking to development experience because one of the challenges is your tests need to run on the native device in the application. So, you have to build the app. You have to push the app into the actual device. So, there's like all the setup of installing the at the app on the device.

CHARLES: You have to launch the simulator.

TARAS: Yeah, right.

CHARLES: To make sure the device is connected.

TARAS: And you run your tests in there. So, that created kind of this situation where we say let's just kind of set Appium aside and just use unit testing which is a very small fraction of the kind of testing that we actually want to do. It will test very little. But let's just do that because getting functional testing to work was really kind of not going anywhere. So once we start doing unit testing, one of the challenges is that it takes like 30 seconds to start your tests. And then, if you for whatever reason, made a mistake, the moment you cancel the build, it leaves, like it doesn't clean up of itself well. So, it leaves processes running in the background. And so now, you spend another like 10 to 15 minutes Googling around for a cookie, "How do you find these processes and stop them?" So, we eventually settled on having a script that does that, but this is the kind of things you have to end up doing because there's a bunch of things that are wired together, but they're not wired together in a way that is seamless. And so, you end up kind of just debugging a lot of stuff where you just want to run some tests but you end up doing all these other stuff.

CHARLES: Right.

TARAS: And you spend a couple of minutes just doing something that you'd expect to happen in like 20 seconds.

CHARLES: Right. There is a feeling that every aspect of the system is coupled to every other aspect of the system in kind of varying ways of interconnectedness. And that's not what you want for a very, very complex system. You want it to be extremely modular.

So, I think we should keep the command line tool. There's probably a separate discussion, I think, about that. But you have to close the book on the Appium and the unit testing. I think the other problem was that you have to run these things on simulators. On macOS, that's not a problem because the simulators ship with X code. And so, you don't actually require an external service. Whereas in CI on Android, it's very unlikely that you're going to have Android emulators on hand because they require a separate virtual machine. Android emulation is actually quite heavy. If you're running through Android Studio or something locally, you essentially need VirtualBox or some equivalent to run your Android simulator because you actually need that simulated hardware. If I understand correctly, that was actually not something that had been really accounted for. It was that you might want to be running simulators not on the same machine as what you were developing on or what the actual that you were building on.

TARAS: Yeah, a lot of the tooling seems to be designed around this idea that you're going to be building and running everything on your machine. And so, you can spin up a virtual machine easily. But in CircleCI, for example, they don't support running a virtual machine inside of a Docker container because for that, you need a feature of a virtualization that is not supported in many CI platforms. You have to run a parallel server if you want to have like Appium running, for example. You need to have a separate server running like an Azure or a Google Cloud somewhere that is able to run virtualized servers that have a host machine that's being guest systems that are running the actual Android emulators of different versions. And so, when I started doing research in this, there are companies that are doing this really well but it's not unusual to be using hardware from Amazon that costs thousands and thousands of dollars per month.

I think for anyone who's getting into mobile development, I would say the hidden gem of Android world is Genymotion. Those that do a lot of Android development, they know about it. But Genymotion has both like a desktop environment and it has SaaS offering that they're in the process of releasing. And so, what it allows you to do is when you run it locally or on your local machine, it allows you to create a virtual machine that is running in VirtualBox and then it allows you to run kind of optimized environment for running Android. And when you do that, it's really fast. It's very smooth. It makes running Android devices locally as easy as it is to run iOS devices on macOS.

CHARLES: I remember starting out and trying to actually just get any Android emulator running on my Mac and I couldn't even do it.

JEFFREY: It was such a huge time saver.

CHARLES: Yeah.

TARAS: And to have this Saas offering is really great because you could basically create your virtual machines on demand and then you install into a virtual machine from your CI server and then you run your tests there. That's kind of the key that I found to be able to run tests and automate it against emulated devices for Android. Genymotion is really great.

CHARLES: Yeah. Again that's the kind of thing that you need when you're in CI. And so, one of the things, I think, one of our discoveries is that there just isn't -- when we started working on this and we haven't seen a culture of running these tools in the cloud and accounting for the fact that you might have not all of the tools running on the same machine.

From, I would say, the beginning, I remember the kind of the diagnostics command didn't work but we were running it on a CI server. So, there's a diagnostics command that you run to see do you have this, do you have that, do you have that. It would work and give meaningful results when I wanted to debug my CI server because when we were initially getting set up, something wasn't building right, there was some dependency missing. And I just wanted a diagnosis but it was trying to install all those tools for me. And I was like, "No, no, no. I don't want you to do anything. I don't want to install them. I'm going to be doing all of that as part of the setup of the CI environment. It's going to be installed, it's going to be cached. I don't want you to just try and like massage my system into a suitable state for NativeScript development. I just want you to diagnose what is wrong. Tell me, am I missing this compiler? Maybe I've got the wrong version of Android SDK. Tell me what's going on." And I couldn't get that to work. That was very frustrating. I think it was because the kind of bulk of the assumptions was that it was going to be individual developers working on their own laptops or their own desktop computers to build, to test, to distribute these applications. I think that's becoming less and less the case. I mean, at this point, that's not a way that we're willing to operate.

TARAS: And we eventually figured out how to do all this stuff, right?

CHARLES: Yeah, we have.

JEFFREY: We have.

TARAS: We have the entire process working but it took a lot longer than one would imagine. It took all the time that we had allocated to it which we thought was very generous amount of time but it took like almost a month to get everything set up. The great part of this is that we do have now everything working. And so, there's a repo where people could take a look if they want to get all stuff working on CI, but it took quite a bit of work in figuring out.

CHARLES: Yeah. Actually, I think worth probably a Screencast to show some of those capabilities because it is really exciting. I mean, when you actually think about the pipeline in its entirety. But we never were able to get functional testing working.

TARAS: And then the challenge here is that because we were essentially looking at NativeScript, going back to this question like, "What do we need to be able to have like hundreds of developers potentially running on this platform?" And so there's a lot of considerations and this tool is just one of them. I think the other one that is a big one is like what are the capabilities of the view layer because that's where most of developers were spending most of their time. We got stuck a little bit about that because I spent a lot of time working in the view layer. The thing that was really great and the thing that I really liked about it is the fact that you have a collection of components that you can use in Angular. You render it as component and then that component is going to look correctly on iOS and is going to look correctly on Android. From a single code base, it's building appropriate components for iOS and Android. What I think is really confusing in that case, though, is because the Android and iOS components don't have parity in a sense. They don't behave exactly the same. And there is also a kind of a reputation in the NativeScript documentation that Android tends to be slower, much slower than iOS. And so, when you start to run into performance problems and you start to run into those pretty fast because it is not really clear what is necessary to not optimize NativeScript, when you start to run into performance problems, it's not really clear like where is it coming from. Right now, the profiling that they have for the UI is very limited. They're kind of in the process of migrating over to chrome.debugger, but profiling in chrome.debugger is not implemented. You can do performance optimization using Android tooling but that's only going to tell you performance of the Java side, or the iOS side is not going to tell you the performance of the code that's running inside of JavaScript. It's not really clear what is causing the problem. If you don't know what's happening, you kind of write it off as like, "I think it's just Android being slow." In reality, when you actually start to dig deeper, you realize there's things about the Android implementation of the components that are different or the views that are different than iOS. And it's the differences that add up to weird performance problems. That's probably the thing that gave me the most hesitation because one of the things that made me think like if we want to be able to give this to a team of like 50 people, we need to have our own view layer because we cannot rely on components. An example of this would be, they have a list ticker on iOS, it doesn't omit change events when you scroll. If the list is moving, it change events and not omit it. But on Android, every time that a different item shows up on a screen, it changes the selection. And so now, you've got this view that's a meeting on Android as a meeting change events. I made an issue around this and the response was that while there's a workaround that you can have for this, but that's hard. Work around is not a solution.

CHARLES: Right. When you have a leaky abstraction like that.

TARAS: Part of the problem is because people use leak abstraction. And so, what's happened in Native -- we actually got on the call with NativeScript core team and they're excellent in really being very helpful, understanding what the problems are, and providing pass on making things better. But what's happened as a result of having this leaky abstraction is that people are relying on the leak. And so now, the leak is the API. And so, we can't change that.

JEFFREY: Right.

CHARLES: And the answer that you really need there is, "We can't change that without breaking stuff. Here's our migration path for deprecating this and introducing a new API." And that gets more into the process stuff and it seems like the process for making changes to the underlying API, I think, could use a little love in the sense that it's kind of opaque as to where the platform is going. There's not a concept of like an [RSC], there's no roadmap about what to expect. What is this API going to look like in the future? Is this stable? If I were writing a software and someone said, "Hey, there's this leaky abstraction," I think my reaction would be, "We've got to fix this." And we also have to acknowledge that there are users who may depend on this. And so, we have to be very deliberate about it.

TARAS: The challenge with this too is that NativeScript kind of outgrew its hands because I think originally, it wasn't meant to be hosting Angular and hosting Vue. Vue didn't exist. Angular didn't exist when NativeScript started. So I think what's happened is that these views that were available, I wouldn't call them components because they don't act like components, but they're exposed in Angular like components but the API feel like Vue objects. So these Vue objects that you consume, that you render in Angular, for example, or in Vue.js, they are the same APIs that NativeScript had before Angular and Vue.js.

CHARLES: Right. You know what? It feels like there's a MVC framework, like a Circa 2010, 2012 MVC framework that has now become the foundational layer for Vue frameworks that have had significant advances in the way we conceive of model in Vue and how data is generated and passed around and how views are rendered off of the data and how reactivity is changed. But there's still, the underlying platform has not evolved. And in fact, this was originally user-facing APIs and now these APIs have become foundational for other user-facing APIs but haven't had the iteration and evolution to make them robust.

TARAS: And flexible enough. As a result, you have the situation where not only is it really super easy to deoptimize the views simply because the requirements of keeping performance expectations are not obvious. One of the things that I found is that the list which is, lists are like 50% of most applications. Before I go into the problem with list, the nice thing about lists in NativeScript is that because they're interacting directly with native APIs, you have really fast list when they're optimized. They're really easy to work with. But they easily get deoptimized by the fact that the expectation to keep the list fast, you have to use this API in NativeScript called array observable and observable. And this is not to be confused with like...

CHARLES: [Inaudible] observables?

TARAS: Yeah.

CHARLES: It's not to be confused, but in fact, every conversation involves a lot of confusion. Because we were using observables, right?

TARAS: And we were actually using observables. So, we're using observable [inaudible] and we're using this array observables and object observables. And so, it's necessary for NativeScript to, essentially what it expects for list to be fast, is it expects that it's going to receive an array observable which is an object that wraps an array because it needs to know when an order or length of data rate changes. So what happens when you pass an array observable, a NativeScript array observable into a list? It will listen for change events on that object. But if you want to change the value of each of the items, like if you want to change a property on the object and have your view remain optimized, the array observable has to have an observable object which allows NativeScript ListView to listen for changes, property changes on the object. You pass this array observable which contains observables that ListView listens for changes on to make sure that it knows how to correctly apply this change to the list. If you don't have this magic, like if you haven't figured out this recipe for ListView performance success, you're going to have a really hard time because it's really not clear at what point and how this thing got deoptimized, why has it just gotten slower.

CHARLES: There's a lot of iteration that needs to happen there and it's not clear what the plan, what the priority, or even how you will even begin to go about this. Because I think that the internal working is that it seems basically to be controlled by one company. I don't recall seeing any contribution from anybody except for Progress which is Progress Incorporated is the company that's kind of the controlling interest, the original company that developed it.

TARAS: The way this showed itself very practically is that to make changes too -- so they have a ListView which comes with NativeScript public and there's RadListView which is the component that has a lot of stuff on it. Like if you want to pull to refresh or if you want to do like laser loading a data or if you want to do a filtering, you want to do -- so most people use RadListView. But RadListView, you can install, so there's no limitation when you build to install it, and your node modules has the source code for that. But the source code, the original TypeScript code, untranspiled code is not publicly available. They have a process for doing this and it's very nice that everybody's very kind and very accommodating. You send an email, they'll give you access to this repo and then you'll have the ability to contribute. NativeScript core team is very helpful and they're open to contributions. There are changes that need to be done to the Angular implementation to make it faster without having to put the requirements of the observable thing, and so they can give you a path to make that stuff happen but it's not open source in the sense that it's not a traditional open source that we would kind of expect. So, there's all kinds of hoops that you need to jump through and the source code is very difficult to read because it's transpiled from TypeScript to JavaScript.

CHARLES: And there was a certain level of opacity in terms of process. For example, I filed an issue which was actually a blocker. For us, it was actually causing our Android build not to work. I didn't hear anything about it. And then, all of a sudden like four days later, a fix came through referencing another repository on which this thing depended with. There was not a lot of context service. So it was obviously referencing a bunch of context that probably happened between two people in a face-to-face conversation. But I couldn't really tell what was going on, why it was an issue, because there was no comment. It was just a pull request that was referencing this issue. I never got a notification. I actually had to go and be like, "Hey, I really would like for this issue to be solved. I wonder if I..." I was actually going to post a, "Hey, is there any progress on this?" Or, "Is there any way that I can help? What can I do to get this looked at?" And I saw that there was another pull request that had referenced my issue. And it was merged and I looked down, but then there was no indication of when this would be available for public release, how I might be able to work around it. And so, the strange loop that didn't get connected was, "Hey, you've got a user who files an issue. You actually use this as the impetus to fix the issue and make a release." But then that whole process was completely invisible to me.

TARAS: You know what? It sounds like you wanted for it to work [inaudible] but you got a pulling mechanism.

CHARLES: Yeah, exactly. Well, I wanted someone to say like, "Hey, here's what's going on, and we're looking right into it." Or, "We're going to look into it in like two months," or, "We can't address this now. But here's a workaround for it." Or, "I don't have a workaround." That's just kind of the expectation that you have when you're playing with open source. In many ways, it does not feel like an open source project.

TARAS: Let's just do a quick note about Saas. Jeffrey, what did you find about the styling of NativeScript views?

JEFFREY: All the components that come kind of shipped as part of the NativeScript core set of components all have styles attached to them. They have CSS attached to them. And as part of the standard data script workflow, with your build toy, you have SaaS available which is very nice. But actually on a recent project, we're not using Saas at all. We're simply using post-CSS and we were able to kick out some CSS variables that turned out to be really nice for theming. So as kind of a future friendly experiment, we were trying to have a light theme and a dark theme since that is very recently now a core part of Android and very likely will be part of iOS this year, where there's kind of a light theme and a dark theme for everything. We were trying to do that. The simplest way to do that with standard web tools is with CSS variables. You can have the flexibility, you have the theming with those. It's so nice. You just, "Hey, my primary color is this color in one scenario and it's this color in another." And we just didn't really have the flexibility to do that with SaaS by itself. And so, that's kind of a limitation of the tooling right now that I hope in the future, we'll have some more sophisticated CSS tools. And really, NativeScript's move toward Webpack and having that as a primary part of the workflow really opens up that possibility that I hope somebody runs with in the near future.

TARAS: Yes, let's bring it all back together.

CHARLES: Can we pause for a moment? Because I actually do think it's important that we at least touch on the command line. I can give a little bit of a kind rant in here but I think that's actually something really important that we have to talk specifically about that.

The other thing that I wanted to touch on very briefly as we kind of draw to the close is the command line tooling, in particular in NativeScript. I think that this is probably one of the weakest points of the platform. And again, I don't want to disparage anybody working on NativeScript. It's an extraordinarily complex problem. This is a command line tool that needs to manage launching simulators, installing things into simulators, pushing code to those simulators. It needs to handle hot updates to things that it's running on, devices and simulators. So, it needs to be building JavaScript assets either with Babel or with TypeScript. It needs to be building those SaaS assets that you were just talking about, image assets. But it needs to be doing all of this for two platforms, so it needs to be managing everything that I just described. It needs to be managing on iOS. Everything that I've just described needs to be managed on Android, as well. It needs to work for a single developer's desktop. It also needs to work with all of those components that I just described distributed out in the Cloud. So, we're talking about an extraordinarily complex piece of software. And I think that unfortunately, the NativeScript CLI does not inspire confidence because it can do all of those tasks.

But Taras, you also mentioned often if you stop the process midway, it will leave a thousand things open and they're just spewing output to your console. The console output, unfortunately, means there's a big noise to signal ratio because it puts out all of the content for Webpack. Every little thing that it's doing with any of the devices, it's logging to the console. So, it doesn't give you a sense of control. So, what you really are looking for in terms of a command line is, "Hey, I've got this incredible sprawl of complexity and I want to feel like I'm on top of it." And unfortunately, by leaving these things open and having so much console output and having the console output not be formatted well, there's all kinds of colors. Every single tool that you're using whether it's Webpack or whether it's Karma or whether it's just console outputs that you are happing inside of your NativeScript application, the brand of those tools comes through. Webpack is a great example. Its console output feels very Webpack. So when you've got Webpack content randomly interleaved with your console content from your Mocha content, from Karma, all of these competing brands, it doesn't feel like a cohesive developer experience. And so, I really, really hope that -- so, to the point being where I felt like I could not live with that command line tool without rewriting it myself. If we want to use this platform long term, we'd have to either have an alternative command line tool or really, really, really help the NativeScript team completely and totally rewrite the command line experience.

TARAS: I would love to work on fixing a lot of these parts about NativeScript if there was a way to actually do it in terms of like, if they wanted to pay us to help them kind of bring some of these things to a state that would match. For example, what's available in Ember or available in React CLI, I would love to do that.

CHARLES: React Native, yeah.

TARAS: Yeah, let's do that work. But who knows what's in store? A lot of awesome platform like the idea around NativeScript architecture is fascinating and it's really, really powerful and really wonderful people doing some, trying to tackle really challenging problems, but it's all glued together in a way that doesn't instill confidence. And it just makes everything feel wobbly, just makes it feel like you never know, is it a problem? Where's the problem from? What is causing this?

CHARLES: Yeah. And if I fix this thing, is it going to break something else?

TARAS: Yeah, we've seen it happen actually with one of the solutions that was introduced to a bug that you were referring to earlier.

CHARLES: Yeah. So that was our three months experience working with NativeScript.

TARAS: We are considering other things now, very seriously looking at Flutter as an alternative for the same client, same scenario. Flutter is looking pretty exciting. There's a lot of things that are really good there. So in three months, we'll do another report and talk about Flutter and what we found. So, that's it.

CHARLES: And I will say I'm actually not like super excited about dart but I'm in dart spot.

JEFFREY: That's a whole other conversation for yet another episode.

CHARLES: I think that, to continue the conversation maybe next week, next time we have kind of an internal podcast, is I would like to really talk about platform evaluation because really you need three months, at least, to get a good idea of this. Is this going to work for the next five years? And most of the time, we give it a week or give it a two week. Or someone comes on who's really excited about this one particular technology and you go off on that tangent. I think there's an interesting meta discussion about how do you select technologies. And we don't have time for that now, obviously. But it's definitely something that I want to have in the future.

TARAS: Sounds good. I think that will be a good conversation for sure.

CHARLES: I guess that is kind of the executive summary on NativeScript from our perspective. With us being three months in, I think, like you said, there's a lot there.

Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at [email protected]. Thanks and see you next time.