Updates from kmitov Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 9:56 pm on November 28, 2020 Permalink |
    Tags: , ,   

    How to do headless specs with the BABYLON JS NullEngine 

    [Everyday code]

    In buildin3d.com we are using BABYLON JS. To develop a headless specs for BABYLON JS that could run in a Node.js environment or without the need of an actual canvas we can use BABYLON.NullEngine. A spec could then look like

     const engine = new BABYLON.NullEngine();
     this.scene = new BABYLON.Scene(engine);

    Here is what I found out.

    Headless specs

    We run a lot of specs for our BABYLON JS logic. All of this specs are against preview.babylonjs.com. The preview version of babylon give us access to the latest most recent changes of babylon that are mode available to the public. These are still not release changes, but are the work of progress of the framework. They are quite stable so I guess at least the internal suite of BABYLON has passed. Preview is much like a nightly build. In BABYLON JS case it is also quite stable.

    2 days ago much of our specs failed. I reported at https://forum.babylonjs.com/t/failure-error-typeerror-cannot-read-property-trackubosinframe-of-undefined/16087. There were a lot of errors for :

    Cannot read property 'trackUbosInFrame' of undefined.

    Turns out that many of our specs were using BABYLON.Engine to construct the scene like

    const engine = new BABYLON.Engine();
    this.scene = new BABYLON.Scene(engine);

    The BABYLON.Engine class is only intended for the cases where we have a canvas. What we should have been doing for headless specs without a canvas is to use BABYLON.NullEngine. Hope you find it helpful.

    Write more specs.

     
  • kmitov 9:39 pm on November 28, 2020 Permalink |
    Tags: , , specs, , web development   

    Why we should never clear our DB before/after running specs. 

    One common “mistake” I’ve seen a couple of times is to clean the Database before/after specs are run. It seems to be a common practice with reasonable arguments. I think this is a bad idea. Here is why and what we should do instead.

    Why is the DB cleared before/after the specs

    When running specs that need access to a DB we might have to create a User or an Article or a Project model, then connect them in a certain way and test the business logic of our spec. After the spec is finished it is not wise to delete these objects from the DB directly in the spec. Sometimes it takes additional time, sometimes it executes additional logic. In most cases you don’t clear the DB after each and every spec.

    It is a good idea to clean the db before all the specs or after all the specs if they are successful. In this way we reset the DB only once, it saves some time and is much cleaner because you can plug in this behavior if you want to.

    Why the DB should not be cleared before/after the specs

    The simple answer is that our code will never, absolutely never work on a clean db in a production. If we have a test procedure that runs the specs against a clean and empty db they might pass when the db is clean. But what use do we have from code that could work in a clean environment, but could not work in a real production environment. The answer is – non.

    We don’t clean our db before/after each spec. In this way we’ve been able to track some really nasty bugs. Like slow queries that are slow only when you have too many users. Other cases involve special relations that are built in time. Like users that are part of an organization and the organization was once having one check for uniqueness of the user and now it has another check. Because the db is not cleared every time we make sure that it is properly migrated with all the needed migrations.

    We found out that a 7 years out test db that is not cleared is closer to a 7 years old production db.

    The test db is not the production db

    The test db is not the production db. It might have the same scheme, that is for sure, but the amount of data in them and the complexity of this data is different. What we need is code that could run on a production db. There is no use of any code that could run only in test environment.

    So here is what we do:

    We export the production db, we change some data like user emails, names and any other sensitive data and we import it as a test db. We run the specs on this db.

    In this way we make sure that the code could actually run on a real db before deploying it.

     
  • kmitov 5:36 pm on November 17, 2020 Permalink |
    Tags: , technical debt   

    Technical (code) Debt and how we handle it. 

    The subject of technical debt is interesting for me. I recently got a connection on linkedin offering me to help us identify, track and resolve technical debt and this compelled me to further write this article and to give some perspective on how we manage technical debt in our platforms and frameworks by specifically stopping on a few examples from the fllcasts.com and buildin3d.com platforms along with the Instructions Steps (IS) framework that we are developing. Hope it is useful for you all.

    What is technical debt?

    Here is the definition on the first source of wikipedia. It is pretty straightforward –

    Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution.

    (https://www.techopedia.com/definition/27913/technical-debt)

    What does it look like?

    The way I think about technical debt is – we write certain structures like for example “if” which in many cases is a Technical Debt. Of course there are many other types, but I will stop at this one for the article.

    // In this example we do a simple if for a step  when visualizing 3d assembly instructions
    // if there is animation for the specific step created by the author and persisted in the file we play this animation, but if there is no animation in the file we create a default animation.
    if(step.hasAnimation()) {
     return new AnimationFromFile(step)
    } else { 
     return new DefaultAnimation(step)
    }

    The problem with technical dept here is that there later in the development of the project two new cases might arise.

    Case 1 – we want to provide the functionality for AnimationFromFile(step) only to paying customers, otherwise we return just the default animation. The code then becomes:

    // We check if the user is subscribed and only then provide the better experience
    if(step.hasAnimation() && customer.hasSubscription()) {
     return new AnimationFromFile(step)
    } else { 
     return new DefaultAnimation(step)
    }

    Why is this bad? Where is the debt? – the debt is that we are now coupling the logic for playing animation with the logic for customers that have subscriptions. This means that when there are changes to the subscription logic and API we must also change the logic for handling animations.

    Case 2 – we introduce a third type of animation that is only for users with a certain WebGL feature in their browser. The code becomes:

    // We check if the browser supports the WebGL feature in question and then return a FancyAnimation.
    if(step.hasAnimation() && customer.hasSubscription()) {
     return new AnimationFromFile(step)
    } else {
     if (webGlFeaturePresent()) {
       return new FancyAnimation();
     else { 
       return new DefaultAnimation(step)
     }
    }

    Now we have a logic that knows about creating DefaultAnimation, about reading from files, about what a subscription is and when are users subscribed and it also knows much about the browsers and their support for WebGL.

    At a certain point in time we would have to refactor this logic and separate it into more decoupled pieces. That is a technical debt.

    How was Technical Debt created (in the example above)?

    We took the easy path now by placing one more if in the logic, but we knew that at some point we would have to refactor the logic.

    Should we omit Technical Debt?

    I think a good architecture could prevent a lot of the technical debt that is occurring. Good decoupled architecture with small units with clear boundaries and no state will result in 0 technical debt and we should strive to create such systems. Practically the world is not perfect. In a team of engineers even if you spend all your time on fighting with Technical Debt it is enough for only one colleague at one instance to take the easy path and add one more if to “fix this in 5 minutes instead of 3 hours” and the Technical Debt is already there. You’ve borrowed from the future.

    How do we track technical debt?

    I have personally learned to live with some technical debt. If I now do

    $ git grep "FIXME" 

    in one of our platforms we would get 37 results. These are 37 places where we think we could have made the implementation better and we even have some idea how, but we’ve actively decided that it is not the time now for this. Probably this part of the code is not used often enough. Or we are waiting for specific requirements from specific clients to come and we would address this them – when there is someone to “pay for it”. Can we address it now? – but of course. It would take us two, three days, but the question is why? Why should we address this now. Would it bring us more customers, would it bring more value to the customers? It would surely make our life easier. So it is a balance.

    Our balance

    I can summarize our balance like this.

    1. We identify part of the code as a technical Debt (because we do regular code reviews).
    2. We try to look at “what went wrong” and understand how this could be implemented better. We might even try in a different branch but we do not spend that much resources on this.
    3. We then know “what when wrong” and we agree to be more careful and not to take debt the next time but to instead implement it in the right way the first time.
    4. After that we decide if it is worth it to refactor the current issue – are the new clients coming that would ask us for modifications on these parts of the code?

    That’s it.

    Simple “FIXME”, “TODO”, “NOTE”, “IMPORTANT”, “SECURITY” tags in the code, git grep to see where we are and balance with trying to learn how to do it correctly next time.

    How can we solve Technical Debt for the example above?

    In buildin3d we have a framework with an event-driven plugin architecture. So for us it was simply a matter of registering a different plugin for the different features.

    // Pseudo code is 
    framework.register(new FancyAnimation())
    framework.register(new AnimationFromFile()) 
    framework.register(new DefaultAnimationExtension())
    
    framework.ariveOnStep((step)=> {
      ...ask all the extensions for animation and play the first animation that is returned
    }})

    The question is at which stage do you invest in a framework.

    What about MVP(s)?

    The greater balance is sometimes between an MVP and a working product. On one occasion we had an open source tool that was doing exactly what we needed. It was converting one 3D file format to a different 3D file format. We started the project. We used the open source tool. We delivered a working MVP in about a month and we took a lot of debt, because this tool came with other dependencies and was clearly not developed to be supported and extended. It was clear from the beginning that once new client requirements started coming we would have to re-write almost everything. And we waited. We waited for about 2 years. For 2 years we were extending the initial implementation and one day a client came with a requirement that we could no longer support. Then, it took us about 6 months to re-write the whole implementation in a completely new, much more extensible way and could easily accommodate new requirements.

    Conclusion

    Try not to take technical debt.

    If you have to then at least try to learn why it happens and learn how not to do it in the future. You will exponentially become better.

    Write down a comment in the code about why do you think this is a debt and how it should be approached. Spend some time reviewing and resolving debts if it pays off.

     
  • kmitov 11:05 am on November 16, 2020 Permalink |
    Tags: , animations, ,   

    We got featured in the BABYLON JS 4.2 release video 

    About two year ago when we started working on delivering 3D instructions and playing 3D animations and visualizing 3D models in the browser. Little did we know than that with version 4.2 babylon js will feature us in their release video. Thank you BABYLON theme.

    I wrote an article on BuildIn3D.com – https://buildin3d.com/blog/buildin3d-featured-in-the-babylon-js-4-2-release-video/

    Here is the release video for all to enjoy.

     
  • kmitov 6:48 am on November 2, 2020 Permalink |
    Tags: , ,   

    A week ago I gave a nice lecture about Google Closure Compiler and how to use it in ADVANCED_OPTIMIZATION mode. It is available in Bulgarian at https://softuni.bg/trainings/3194/advance-javascript-compilation-with-google-closure-compiler-why-and-how

     
  • kmitov 6:57 pm on October 31, 2020 Permalink |
    Tags: startup   

    From MVP to first support Ticket – 1 month 

    1 month. That’s how long it took for clients to move from “let’s try your service” to “I need support. Why is this not working for my very special case? I love it and I want this service!”

    1 month. That’s how long it took from starting the servers that are running BuildIn3D (https://www.buildin3d.com) to the first support request.

    Users come and try the service. Upload a few hundred instructions. Errors occur and we fix them but no user ever sends a support request. It took one month until someone is brave enough to ask for support. I guess that in the begging people just don’t believe your service is actually working and it takes about a month for them to get from “let’s just try it out” to “I demand this should work. I need it.”.

     
  • kmitov 4:25 pm on October 23, 2020 Permalink |
    Tags: , mandarin, utf, wget   

    "wget 1.20 is here!" (never in my life I though I'd say that) 

    (this particle is part of the Everyday Code series)

    There are tools that just work. Low level tolls doing much of the heavy lifting and one thing you know about them is that they work. You never write

    
    $ mv --version source target
    

    For that matter you also never write

    $ cp -- version 1.3 source target
    $ ssh --version 1.2.1 use@machine
    $ curl --version 773.2  url

    Same applies for wget. There are tools that always work. Because they do one thing and they do it well. Not that there are no version. There are. But today was the first time in my career that we had to upgrade a wget version. We moved from version 1.17 to 1.20

    Why the change

    What could have changed that made it important to update from 1.17 to 1.20?

    It was the introduction of support of some mandarin symbols. Mandarin. A language. Users were uploading files names with such symbols and we had to support them.

    The internet. What a beautiful place.

     
  • kmitov 5:54 am on October 21, 2020 Permalink |
    Tags:   

    How I became one of the top JavaScript developers I know of? 

    TL; DR; How do you measure how good you are? – my highly unscientific measurement is around “when others start asking and listening to you” and when you “start delivering faster and better results”. I became really good in three steps – start solving a really hard problem, ditch the JavaScript dependencies coming from the community and pack everything.

    Context

    If you’ve been in the field of Software development and you are already a good developer and you’ve used but are generally avoiding JavaScript (because reasons), then this article is just for you. It sounds oddly specific, but in the same time I am constantly seeing many people in this situation. This was me a few years ago so I hope it could help you.

    In about 1.5 years I’ve become one of the best JavaScript developers I know of. The moment I personally started delivering better results than the much more experienced contractors I was hiring, while I was also managing our organization, I knew I have moved a long path.

    What was wrong with JavaScript?

    Nothing in particular and everything. We’ve seen countless articles from well read and published software developers over the last two decades about why JavaScript was “bad”. I am not going to get philosophical here, because I found out that they are all true and yet what stands behind the notion of JavaScript in the beginning of the third decade of the 21 century is still a very vibrant and live community achieving incredible results.

    Step one. Find a difficult real-world problem.

    My problem was to build an event-driven plug-in architecture in JavaScript while incorporating and refactoring legacy code that became one of the extensions. This was version 6 of our Instructions Steps Framework delivering the 3D assembly instructions at buildin3d.com. We are currently working about a hundred extensions.

    Finding a difficult problem could really focus you. It was not difficult from an algorithmic or mathematical point of view. It was not difficult as a new blockchain or AI or another emerging technology. It was difficult from an architectural point of view. How do you build a framework for plugins that live in a browser and that must incorporate some legacy behavior.

    I guess finding a difficult problem applies to learning all kinds of technologies.

    Step two. Ditch the dependencies. Go vanilla.

    The community is large. We have JS in the browser, in native apps, on the server. We have JS everywhere. The entry barrier is extremely low. You can build a new JS tool today, publish it on Reddit and have hundreds of people trying it tomorrow. In half a year your have hundreds of npm packages that are dependent on you, you change something and the whole npm is broken. This happens more often than you might think.

    Most of the solutions that could be found on the internet are outdated or plain wrong. Most of the solutions are not about the problem that is addressed. I guess because of the size of the community there are so many “answers and solutions” on the internet around – “take this library and do this call”, but these are not solutions for the case you are working on and are solutions that surely will not work in a year or two.

    Example. I’ve never in my live done something like:

    $ mv --version 1.2 source.md target.md

    This is a command for moving the file source.md to target.md where I’ve specified the “version” of the mv that is to be used. Does this sound familiar? No, of course. I am not sure even if it is possible to specify the version of mv to be used. I had an enormous difficulty explaining this to the maintainers of some of the popular tools used in the JS world and they did not agree with me. (No link here because I don’t want to point fingers.)

    I can not forget an interesting lecture by Linus Torvalds that I once watched where he was:

    You never break libraries ABI (Application Binary Interface). This is the thing I am very serious about in Linux Kernel. You don’t break ABI. You support it.

    But that is not how much of the JS community and in fact many other communities think.

    So I ditched the dependencies. I went full scale vanilla JS. I was not even depending on the DOM for much of the implementation and the things that were dependent on the DOM and some of the WEB APIs were extracted into proper extensions. Without any dependencies I had to do more work, and I would admit I had to write a tree traverse and observers on my own which I never expected I would have to. But at the end this ends up to be trivial implementations extracted to a proper extension that we could easily change with a new more well maintained library (when I find one).

    Step three. Pack and Go.

    Webpack it is. There are others. I’ve used many tools and I found out it is extremely important to pack your JavaScript properly and to understand the whole process. So look at the tool for packaging and try to develop the JS into different modules but pack them all into a single file that you deliver to the browser. The amount of issues you could face with this is really astonishing.

    Step four. Debug a jQuery problem.

    It took me weeks to migrate one of the platforms that we are running from jQuery 1 to jQuery 3.5. The amount of work required to completely remove jQuery dependencies was still too much so we decided to first migrate it and probably later remove it. Our code was not dependent on jQuery1, but some of our dependencies were. I have a friend that is now migrating from jQuery 3.4 to jQuery 3.5. I call these things the “jQuery problem” – as the library made it so easy to approach and solve different problems it was used in all kinds of unintended ways that it was never designed to. As the answers in blog posts and internet forms were based mostly around jQuery there was a decade or two with a lot of copy/pasting going on. Debugging such hard problems could be an eye opener.

    Conclusion

    A lot of other things matter – like choosing the right tools. But in retrospect these were the things I learned most from.

     
  • kmitov 4:22 am on October 21, 2020 Permalink |
    Tags: ,   

    Advanced compilation with Google Closure Compiler 

    Tonight I am doing a live stream lecture about Google Closure Compiler and how we use it to compile the JavaScript code of the Instructions Steps Framework. The framework is the core of buildin3d.com where we deliver 3D assembly instructions.

    The repo for the lecture is located at https://github.com/thebravoman/google-closure-compiler-presentation/.

     
  • kmitov 5:38 am on October 19, 2020 Permalink |
    Tags: management,   

    Don’t fix the issue in the software. Improve the process. 

    Yesterday one of the features on our platform did not work. I was in a meeting, demonstrating it over a shared screen and talking with a potential client. I went to the page showing the IS Editor in our buildin3d.com platform and the editor for editing the assembly instructions did not start. A little rush of embarrassment and a few milliseconds later I knew what I had to do. Thanks to my seniority and extended experience in the world of web development I moved my fingers lighting fast on the keyboard and I refreshed the page. The editor started. The demonstration continue.

    I could remember that I stumbled upon this issue a few days earlier and I saw that the IS Editor was not loading when you first visit the page. The meeting continue, I said something like “Sometimes when we are sharing the screen my bandwidth is small so we have to wait”. I suppose the client did not exactly understood what has just happen, but what I know is that the next time they try it on their side it will not work and they will be disappointed.

    Right after the meeting there was a problem I was facing. Should I now open the repo and start debugging or should I wait a day or two for our team to look at this.

    One of the most difficult things running a Software company as a good software developer is the patience to wait for the team of developers to resolve an issue.

    I was close to mad. How difficult could it be? After you commit something just go to the platform and see that it works. We have a lot of automation, a lot of testing and spec that have helped us a lot. We have a clean and I would say quite fast process for releasing a new version of any module to the platform. It takes anywhere from 2 minutes to about 20 minutes depending on what you are releasing. So after you release something just go and see and test and try it and make sure it works. How difficult could it be?

    I was mad. Like naturally and really mad. Not that this demonstration was almost ruined by this issue. I was mad that we’ve spend about 3-4 months working on this editor and it currently does not start. It is not true that the editor itself is not working. It is just not starting. Once it starts it works flawlessly, but a mis-configuration in the way it is started prevents it from even starting.

    It’s like getting to your Ferrary and it does not start because of law battery on your key or something. There is nothing wrong with the Ferrary itself, but your key is not working.

    In this state of anger I opened up the repo. I tracked down the moment it was introduced. And here is the dilemma:

    1. Should I now start debugging it, and resolving it?
    2. Should I just revert the last 11 days of commits and return the platform to a previous state completely removing the great improvements we’ve introduced in this last 11 days?
    3. Should I leave it for the next few days for the team to look at?

    The worst part is that I can fix the issue myself. But that is not my job. My team counts on me to spend more of my time with potential & existing clients, talking and discussing with them. Looking for ways they could integrate us. But in the same time I had an issue where a major feature is not working and will not work for the next few days and in one sleepless night I could resolve it.

    I don’t have this problem with the other departments. When there is an issue with some of the 3D product animations and models or there is an issue with some of the engineering designs I do not feel the urge to go and resolve this issue. I have the patience to rely on the team for this. Basically because I lack the knowledge and the tools to resolve such issues.

    Years ago when we were starting with 3D animations and models I had great interest, but I openly refused to install any software about 3D animations and models on my machine. I knew myself and I knew my team. In school an in university and was trying some 3D models and animations and it felt great. I learned a lot and I had some great time working on such projects. So I knew that if I install some of the software on my machine there will be issue that will come to me, but that was not my role in my organization.

    Same for engineering. I have the complete patience to wait for days for an engineering design task to complete. I never start the SOLIDWORKS myself and go on and “fix the things”. I could. I just don’t want as it will distract me from other important things and I know I can count on the engineers to do it.

    But with software it is always a little difficult. Not that I can not delegate. I can. There are large parts of the code we are running that I have never touched, or changed or anything. So I though – why was this particular issue different? What was my problem? Why was it bothering me? Why was this different from any other issue in software development that is reported, debugged and resolved. Where did the anger come from?

    I was angry because the process I’ve setup has allowed for this issue to occur.

    The IS Editor was working a few days ago. Now it was not working. This was not an issue of my software development skills, this was a challenge for my “organizing a software development process that produces a working software and deploys it to production a few times a day in a team with a large code base and a new R&D challenge that we were working on”.

    This I have found in my experience to be the most difficult problem for good software developers that mediocre and bad software developers do not face. When you know how to fix it, how to implement it and you take on the task then your time and energy is spend on resolving the issue. It might be better for the team as a whole if you spend your energy and resources on a different tasks – like how to avoid a regression in a multi-teams multi-frameworks environment.

    Know what is important and where your efforts would be most valuable. I’ve stepped up and did a lot of software development int he team. I’ve single-handedly implemented a number of frameworks. Not just the architecture, but actual implementation. I once deleted two human years of development and re-implemented the whole module almost from scratch. There is even a saying in the team “Kiril will roll up his sleeves and will implement this”.

    But no.

    There will always be issues in software development and we should think if our task is to resolve this issues, or to make sure this issues never occur in the first place. The later is objectively the more important and difficult task.

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel