Updates from October, 2020 Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 5:54 am on October 21, 2020 Permalink |
    Tags:   

    How I became one of the top JavaScript developers I know of? 

    TL; DR; How do you measure how good you are? – my highly unscientific measurement is around “when others start asking and listening to you” and when you “start delivering faster and better results”. I became really good in three steps – start solving a really hard problem, ditch the JavaScript dependencies coming from the community and pack everything.

    Context

    If you’ve been in the field of Software development and you are already a good developer and you’ve used but are generally avoiding JavaScript (because reasons), then this article is just for you. It sounds oddly specific, but in the same time I am constantly seeing many people in this situation. This was me a few years ago so I hope it could help you.

    In about 1.5 years I’ve become one of the best JavaScript developers I know of. The moment I personally started delivering better results than the much more experienced contractors I was hiring, while I was also managing our organization, I knew I have moved a long path.

    What was wrong with JavaScript?

    Nothing in particular and everything. We’ve seen countless articles from well read and published software developers over the last two decades about why JavaScript was “bad”. I am not going to get philosophical here, because I found out that they are all true and yet what stands behind the notion of JavaScript in the beginning of the third decade of the 21 century is still a very vibrant and live community achieving incredible results.

    Step one. Find a difficult real-world problem.

    My problem was to build an event-driven plug-in architecture in JavaScript while incorporating and refactoring legacy code that became one of the extensions. This was version 6 of our Instructions Steps Framework delivering the 3D assembly instructions at buildin3d.com. We are currently working about a hundred extensions.

    Finding a difficult problem could really focus you. It was not difficult from an algorithmic or mathematical point of view. It was not difficult as a new blockchain or AI or another emerging technology. It was difficult from an architectural point of view. How do you build a framework for plugins that live in a browser and that must incorporate some legacy behavior.

    I guess finding a difficult problem applies to learning all kinds of technologies.

    Step two. Ditch the dependencies. Go vanilla.

    The community is large. We have JS in the browser, in native apps, on the server. We have JS everywhere. The entry barrier is extremely low. You can build a new JS tool today, publish it on Reddit and have hundreds of people trying it tomorrow. In half a year your have hundreds of npm packages that are dependent on you, you change something and the whole npm is broken. This happens more often than you might think.

    Most of the solutions that could be found on the internet are outdated or plain wrong. Most of the solutions are not about the problem that is addressed. I guess because of the size of the community there are so many “answers and solutions” on the internet around – “take this library and do this call”, but these are not solutions for the case you are working on and are solutions that surely will not work in a year or two.

    Example. I’ve never in my live done something like:

    $ mv --version 1.2 source.md target.md

    This is a command for moving the file source.md to target.md where I’ve specified the “version” of the mv that is to be used. Does this sound familiar? No, of course. I am not sure even if it is possible to specify the version of mv to be used. I had an enormous difficulty explaining this to the maintainers of some of the popular tools used in the JS world and they did not agree with me. (No link here because I don’t want to point fingers.)

    I can not forget an interesting lecture by Linus Torvalds that I once watched where he was:

    You never break libraries ABI (Application Binary Interface). This is the thing I am very serious about in Linux Kernel. You don’t break ABI. You support it.

    But that is not how much of the JS community and in fact many other communities think.

    So I ditched the dependencies. I went full scale vanilla JS. I was not even depending on the DOM for much of the implementation and the things that were dependent on the DOM and some of the WEB APIs were extracted into proper extensions. Without any dependencies I had to do more work, and I would admit I had to write a tree traverse and observers on my own which I never expected I would have to. But at the end this ends up to be trivial implementations extracted to a proper extension that we could easily change with a new more well maintained library (when I find one).

    Step three. Pack and Go.

    Webpack it is. There are others. I’ve used many tools and I found out it is extremely important to pack your JavaScript properly and to understand the whole process. So look at the tool for packaging and try to develop the JS into different modules but pack them all into a single file that you deliver to the browser. The amount of issues you could face with this is really astonishing.

    Step four. Debug a jQuery problem.

    It took me weeks to migrate one of the platforms that we are running from jQuery 1 to jQuery 3.5. The amount of work required to completely remove jQuery dependencies was still too much so we decided to first migrate it and probably later remove it. Our code was not dependent on jQuery1, but some of our dependencies were. I have a friend that is now migrating from jQuery 3.4 to jQuery 3.5. I call these things the “jQuery problem” – as the library made it so easy to approach and solve different problems it was used in all kinds of unintended ways that it was never designed to. As the answers in blog posts and internet forms were based mostly around jQuery there was a decade or two with a lot of copy/pasting going on. Debugging such hard problems could be an eye opener.

    Conclusion

    A lot of other things matter – like choosing the right tools. But in retrospect these were the things I learned most from.

     
  • kmitov 7:07 am on September 30, 2020 Permalink |  

    ApplicationController.render – one more way for a CSRF token to annoy you. 

    CSRF token and ActionController::InvalidAuthenticityToken” could be some of the most annoying issues in a rails application. Here is one more way you could get annoyed, and it is rarely the rails fault.

    ApplicationController.render

    The method allows us to render rails partials outside of the views and controllers. It is useful for things like rendering a view in a delayed job. There is no request context there, no session and yet sometimes you would like to render a view in a job.

    I think it was introduced in Rails 5 and we’ve been using it here and there.

    ApplicationController.render for a json response.

    We are using Datatables-net and we generate a JSON response that contains some HTML that should be visualized to the user. The HTML is a button_to for a delete form. In the table for our groups we had a remove action. This remove action is actually a form for an HTTP delete request.

    The form has a token

    It turns out that the token generated in the context of ApplicationController.render is invalid. There is no session passed to ApplicationController. We should have just used ‘render’ – the method available in the views.

    The correct fragment was

       def generate_remove_button_for record
    -    rendered_string = ApplicationController.render(
    +    rendered_string = render(
           partial: 'shared/fc/admin_unify/table_actions',
    +      formats: [:html],
           locals: { 
             links: get_edit_delete_actions([*@namespaces, record]),
             id: record.to_param
    
     
  • kmitov 1:03 pm on April 22, 2020 Permalink |
    Tags: , jquery, , , , ,   

    Rails 6 + webpacker + jquery + sprockets + jquery plugin (fancetree) 

    So you might be in the process of migrating to webpacker. This means your sprockets should continue working. This is difficult. Sprockets wants jquery available in the view and you don’t have jquery available in the views. You have it in the webpacker packs.

    These things won’t work

      <script> 
        $("element_id")
      </script>

    jQuery is only available to the PACKS it is not available to the VIEWS.

    But there is a solution – expose-loader

    Here is how to setup jquery to be available to the views in sprockets app that you are migrating to rails 6. I am starting from the previous article were we set up the project from 0. – https://kmitov.com/posts/rails-6-webpacker-yarn-fancytree-less/

    $ yarn add expose-loader

    Add configuration for exposing of jquery

    // config/webpack/environments.js
    const { environment } = require('@rails/webpacker')
     
    const less_loader= {
     test: /\.less$/,
     use: ['css-loader', 'less-loader']
    };
    environment.loaders.append('less', less_loader)
    
    +
    +const webpack = require('webpack')
    +// this makes jquery available in all pack and you don't
    +// have to import or require it each time 
    +environment.plugins.prepend(
    +  'Provide',
    +  new webpack.ProvidePlugin({
    +    $: 'jquery',
    +    jQuery: 'jquery'
    +  })
    +)
    +
    +// this exposes jquery to be available in the views
    +// <script>
    +//   console.log($('#tree'))
    +// </script>
    +environment.loaders.append('expose', {
    +  test: require.resolve('jquery'),
    +  use: [{
    +    loader: 'expose-loader',
    +    options: '$'
    +  }, {
    +    loader: 'expose-loader',
    +    options: 'jQuery',
    +  }]
    +})

    Also expose fancytree

    // config/webpack/environments.js
    ...
    +// this exposes fancytree to be available in the views
    +// <script>
    +//   console.log($('#tree').fancytree())
    +// </script>
    +environment.loaders.append('fancytree', {
    +  test: require.resolve('jquery.fancytree'),
    +  use: [{
    +    loader: 'expose-loader',
    +    options: 'fancytree'
    +  }]
    +})
    

    And you are done.

    How in your views you could do:

    <script>
      console.log($('#tree'))
      $(function(){
        $('#tree').fancytree({
          extensions: ['edit', 'filter'],
          source: [
            {title: "Node 1", key: "1"},
            {title: "Folder 2", key: "2", folder: true, children: [
              {title: "Node 2.1", key: "3"},
              {title: "Node 2.2", key: "4"}
            ]}
          ],
        });
        const tree = fancytree.getTree('#tree');
        // Note: Loading and initialization may be asynchronous, so the nodes may not be accessible yet.
      })
    </script>
    

    Fancytree as a jquery plugin is working in rails 6 views and is available also to sprockets compiled files.

     
  • kmitov 4:43 pm on April 1, 2020 Permalink |
    Tags: documentation,   

    This code is: ‘BAD’. This code is: ‘GOOD’ 

    This article is part of the series [Everyday Code].

    TL; DR;

    When providing examples in developers documentation always add the bad example, but make it stupid clear that “this is a bad example and it should not be done like this, while this is a good example and it should be done like this!”

    Too long; but I will actually read it.

    We are copy/paste developers. We are fighting challenging problems on tight schedules with complex frameworks running on enormous stacks and we rarely use tools designed with ergonomics in mind. When we stumble on a problem we go for an internet search, copy the first result that looks like a solution and continue. (this is also my main mode of work with nginx. I love it, but it is always so much work to get this assets through a proxy, and this server was even designed for this.). Or AWS API…

    Yet, we somehow manage to deliver software platforms and most of them are in a running state. Brilliant.

    One thing that we are missing in the documentation is of course the understanding of the “why”. Why things are implemented like this? What were the considerations? Why it should not be implemented in different ways? You get this in a book, but you don’t have it in the documentation.

    Most of the time of course we don’t have the time to enter into the details, but I would make the argument that the more we understand the underlying problem, the more fun it is and the more satisfaction that we can get from work. Because most of the time we just can get not satisfaction. (no reference intended, but here it is – https://youtu.be/EU1t1PqASuQ?t=47)

    But this brings us to the problem. When developing a tutorial for how to use your framework should you include the bad examples along with the good?

    Should you include the bad examples in the documentation?

    This was an issue in the early stage of our Instruction Steps Framework developed at Axlessoft.com. As a proper framework it has documentation. As it is a framework it puts some boundaries and rules on what you should do. But we’ve tried to go one step further and describe why we are doing it like this.

    I’ve found it rewarding to include information about why specific decisions were taken not only what are these decisions.

    Example 1 – we’ve decided not to have a central object model on which all extension are based. A central object model would simplify a lot of things and is a lot easier for many developers, but it is less flexible.

    Example 2 – we’ve decided that certain event methods could return Promises, but could also not. Now why would you do this to yourself is a really good question?

    In these two and other examples I have taken the time to include proper documentation and tutorials on why we are implementing a feature like this. But I failed at one place. I was including both the bad and the good examples in the documentation.

    Looks at the example below for iterations from the previous article. This is an example for iterating over trees.

    anArray = node.getDescendants();
    // here you have all the elements of the tree in anArray
    
    // loop over the array
    anArray.forEach((a)=> {
       if(a.name.includes("1")) {
          print a.getDescendants().length
       }
    })

    Here is an example without getDescendants()

    anArray = []
    rootNode.traverse((node)=> {
       anArray.push(node)
    })

    See what I did here. If you came to this article looking for ways to iterate over trees it is completely unclear what is good and what is bad. Writing it right here in the text as “the first example is bad”, “the second example is good” won’t help. Who is even reading this text? So you arrive at the article, get the first iteration that is “bad iteration”, copy it and paste it in your code.

    The point is – it is valuable to include the bad examples as they give people the understanding why the good examples are good. But by including the bad example a large percentage of your readers will just copy paste the bad example and continue.

    My solution

    Mark your examples clearly in the code as “GOOD” or “BAD”. Not in the text. We are developers, we don’t read text.

    // BAD EXAMPLE - because it uses a lot of memory
    anArray = node.getDescendants();
    // here you have all the elements of the tree in anArray
    
    // loop over the array
    anArray.forEach((a)=> {
       if(a.name.includes("1")) {
          print a.getDescendants().length
       }
    })

    Again some text that no developer is ever reading, as it is in between the examples.

    // GOOD EXAMPLE
    anArray = []
    rootNode.traverse((node)=> {
       anArray.push(node)
    })

    See how clear this is. I dare any developer to now copy the bad example in their code. Hope this approach could help you in time of need.

     
  • kmitov 6:17 pm on March 31, 2020 Permalink |
    Tags: algorithms, , , tree   

    Everyday Trees in Computer Science – [Everyday Code] 

    The goal of this article was to equip developers with everything they need to know about trees to use Instruction Steps Framework (IS) and Instruction Converter (IC) that we are currently working on in Axlessoft. While developing the article for internal use I realized that these are things that every developer should have under their sleeve to successfully use tree structures in their day to day tasks.

    image of a tree that will make the article more interesting. Not a real programming tree

    Some of us might be tempted to revise the heavy Knuth books when it comes to solving the next tree problem. But generally we don’t need to do it. You know the meme:

    • First year Computer Science – “algorithms, data structures, etc …”
    • Second year – “functional programming, object oriented, etc…”
    • Last year – “cryptography, network programming, design of languages, etc
    • First day at work – “move this button to the left”

    So let’s start with a few things you need to know day to day.

    You will see “parent->children” more often than “node->left,right”, eg. binary trees.

    If you have a structure of parent and children this is a tree structure. You have three scenarios:

    1. The parent knows about the children and the children know about the parent. There is a parent.children and child.parent
    2. The parent knows about the children, but the children do not know about the parent. There is parent.children, but there is no child.parent
    3. The parent does not know about the children, but the children know about the parent. There is no parent.children, but there is child.parent

    In many books, algorithms and generally articles you will have people talking about Binary Trees. Now, it might be because of my line of work, but:

    1. I can not remember the last time that I had to properly work with a binary tree – something more than “just find this node”.
    2. For every time I have to work with “binary trees” there are at least 1024 times I have to work with “parent->children” structures.

    Yes, it is the same. But it is also not the same.

    Take away:

    All the three cases represent a tree. It is best to have case 1, but this is not always possible. You will deal with “parent->children” more often than with “left node”, “right node”.

    If there is a library for it, use a library.

    This should be a no-brainer, but a lot of times we like to code a few iterations here and there and develop things for basic iteration over a tree. Just don’t. If there is something tree related that you can not find a library to do for you in about 5 minutes internet search, then you are doing something worth of a Phd or you are searching with the wrong queries (or you are trying to do something stupid!).

    There are of course exceptions. Because when we, as great developers, implement Pre and Post order iteration we are doing it better than anyone else. So this exception is acceptable. Of course.

    Iteration -Pre, Post, In and visiting a node

    Visiting a node means that you are processing it. For example printing to the screen.

    There are a few basic ways. Just search for them on the internet. There are a lot of really nice resources of how you iterate over tress. The basic ways are:

    1. LRN – Left Right Node – first you visit the left child, then you visit the right child, then you visit the node. But as we say this is the theory. In a real life scenario with “parent->children” you visit children[0]..children[n] and then the parent
    2. LNR – Left Node Right – How do you do this for “parent->children” I don’t know.
    3. NLR – Node Left Right – first the parent then children[0]..children[n]
    4. NRL – Node Right Left – first the parent then children[n]..children[0]
    5. RLN – Right Left Node – first children[n]..children[0] then the parent.

    Avoid recursion

    Yes, you have trees. But avoid recursion. Use “traversers”. Almost any reasonably useful library has a “traverser” support. The idea is simple – somebody else is doing the recursion for you and this somebody else is the library.

    In pseudo code:

    rootNode = ... 
    rootNode.traverse((node)=> {
       print node
    })

    The idea is simple. You have a rootNode. It has a method ‘traverse’ that will traverse the whole tree, iterate, recurse, do a lot of magic, but at the end it will traverse the whole tree and will call the “print node” for every node in the tree.

    There is no recursion or iteration in your code. There is just traverse. Sometimes it is called “visitor” ( Visitors pattern, Gang of four, things like this). I kind of preferred “visitor” in early stage of my software development career, but now I more often call them “traversers”. Traversers are visitors, but not all visitors are traversers from my point of view. Would be happy to talk about this over a beer. Anyone..?

    Take away:

    If you are using a library that does not have traverse you should seriously consider using another library.

    Avoid ‘getDescendants()’ and getDeepChildren()

    Yes. These are some libraries that try to help. Imagine that you nave:

    node 1
      node 1.1
        node 1.1.1
      node 1.2
        node 1.2.1
        node 1.2.2
      node 1.3

    You need to get all the children of node 1 and all of their children and you need this in an array. Why? Because reasons. This is the feature request. There are libraries like BABYLON.js that try to help. They offer a “getDescendants()’ method. You can call:

    anArray = node.getDescendants();
    // here you have all the elements of the tree in anArray
    
    // loop over the array
    anArray.forEach((a)=> {
       print a
    })

    This method saves a lot. You do not need a “traverser”. You don’t need to know anything about order. You need all the elements of the whole tree so get an array. But there are drawbacks.

    1. It could be a large tree. So you are creating a copy, but this time in an array and memory and user experience are expensive.
    2. Even if it is not large you are still creating an array. Do you really need an array or you could just traverse the whole tree and do your job without previously creating an array?
    3. If you have to call getDescendants() more then once, you should refactor your code.

    Consider the following:

    anArray = node.getDescendants();
    // here you have all the elements of the tree in anArray
    
    // loop over the array
    anArray.forEach((a)=> {
       if(a.name.includes("1")) {
          print a.getDescendants().length
       }
    })

    In the example above we would like to print the number of descendants for each node that includes the string “1” in its name. How many times is the tree traversed? How many arrays are created? The answer is “a lot”. You traverse once for the rootNode and then for every other node again and you create new arrays.

    Take away:

    If there is a helper method it is probably giving you, the developer, something, but it is at the expense of the user experience – more memory, slower execution. Consider the implications of using helper methods. They might help you, but are they helping the user using your software.

     
  • kmitov 7:47 am on February 21, 2020 Permalink |
    Tags: , ,   

    Feedback could come in any form, not just the one that makes it easy to accept it – [Everyday Company Culture] 

    Culture is how we do stuff around here. Read more about the [Everyday Company Culture] series.

    Today’s resolution. TL;DR

    Feedback from the team could be returned in any form, not only formally, not only verbally, not explicitly, but by the day to day mistakes that we make. Accept all kinds of feedback. Don’t be a “douchebag”.

    Today’s case

    There was a requirement for everybody to write the text “Result” as a first line in the comment of his/her work when marking a task as completed. People were following the rule, but writing “reuslt”, “results”, “Done:”, “Restult”, “Release:” and many others, but were not writing “Result”.

    Instead of pushing for everybody to use the right “Result” at the end we ended up with a list of possible “results” that team members could write as the first line of their result description. It turn out to be pretty funny to add new results and as of today this is the list of all the possible values, accepted by the platform.

    possible_results = ["result", 
      "results", 
      "result:", "results:", #These are added because of Ivan
      "rrrrrr", 
      "rrR", 
      "blabla result", 
      "result blabla", 
      "Special Result Line", 
      "The Result:", 
      "Release:", 
      "Release", 
      "Take this result Kiril, I hope you are having a great time",
      "Resutl", # this one is added because of Mihail
      "Restult", # this one is added because of Mihail
      "Ready:", # this one is added because of Mitaka
      "Ready", # this one is added because of Mitaka
      "Done:", # this one is added because of Aleks
      "Done", # This one is added because of Aleks
      ]

    These results are accepted in any form of case – upper and lower. We have a team member that submits his results as:

    TaKe ThIs ReSuLt KiRiL, i HoPe YoU aRe HaViNg A gReAt TiMe

    Which is arguably, very funny.

    Root cause

    Feedback was not accepted. 

    We as a team all accepted that we should have “Result” of the work and we should have a description what is actually the result. Generally the result is different from the goals of the task. The goal is “develop a feature”. But the result is “visit this URL to see the feature”.

    We were just making mistakes when marking the result and I was pushing for the use of the correct word which is:”Result”. Nobody returned an explicit, verbal feedback – “I think we should make the result with ‘Done:’, because I don’t like writing ‘Result:'”. Nobody said this. We all understood that there should be a rule and we should follow the rule. But we were just making mistakes.

    Full Story

    I was trying to establish a rule in the team – when a piece of “work” is completed you must provide a short description, URL link, commit, picture/video of the new feature. This makes it extremely easy to generate release notes that we make public to the client. Only rule was

    “When you submit your the description of the result the text must start with ‘Result’.

    Something like this will be enough

    Result

    We have implemented a new feature that shows a progress bar when downloading the assembly instruction.

    Here is how it looks like

    https://s3.amazonaws.com/fllcasts/content_pictures/pictures/000/003/479/4d609c2301bbe94f7a1c9825b7a2d85275c18537IS-Showing-Progress.png?1581586917

    After the result is provided we use different internal tools and I could easily generate public client Release Notes for our products and platforms. Examples at: https://www.axlessoft.com/release-notes or http://fllcasts.com/release-notes

    Sounds easy. Only thing you have to do is start your description with “Result”, because other values could be “time track:”, “Code Review”, etc. It took me weeks before I finally realize that we should not be strict on the value “Result”. We should be strict on the behavior that everybody should write a result, but if you happen to make a mistake and write “resTult” – that should not be a problem. We got you on the platform end, accepting different input values for result. Does not matter how you call it, as long as it is there.

     
  • kmitov 6:48 am on August 12, 2019 Permalink |
    Tags: , , ,   

    ‘if-else’ is Forbidden – why and how to reduce logic branching in your code. 

    Each time there is an if-else logic in your code there is some logic branching. A state is checked and based on the result from the check different path are taken. This means that two scenarios are implemented and two scenarios should be support from now on. This could quickly become a problem because the number of supported cases grows exponentially.

    The more decisions you make in the code the more cases you must support.

    At the end you start from Process1 but you have 4 different cases and path that you should support. These are 4 different tests that should be written and these are for different places where things could go wrong.

    The four different paths are Process 1,2,4; 1,2,5; 1,3,6; 1,3,7

    Demonstration of a live example and a real production bug

    We have Subscriptions in the platform. Each subscription has a max_users field that shows how many users you could add to this subscription. If max_users=1 this means you can have only one user using the subscription.

    When adding people to the subscription you have two cases

    1. You successfully add someone to the subscription and you show a message “Success”
    2. The max users is reached and you show a message “Error”

    The code in a simplified manner looks something like this:

    if subscription.save 
       form.show_message "Success"
    else 
       form.show_message "Error"

    While developing we’ve changed the code for the error from form.show_message “Error” to modal_dialog.show_message “Error”

    After that we’ve changed the implementation further and the code for modal_dialog.show_message “Error” was no longed called.

    As a result when the use tries to add someone to his subscription, for which they’ve payed, the app freezes and nothing happens. There is no error displayed and no user is added to the subscription.

    The bug occurred because with the latest changes we’ve forgot to manually check the second case of the if-else clause and there is was not test developed for this case.

    How to remove the if-else clause from this code

    subscription.save
    message = subscription.get_save_message
    form.show_message message

    The subscription.save knows what message to set based on whether the subscription was successfully saved. It could set Error or it could set Success. After that we just show whatever message the subscription has to the form. If it is empty that’s great. This means no errors have occurred. subscriptions.get_save_message could be implemented in many different ways. It could be on the subscription object or another object, but this depends on the technology and framework used. But after the method save is called the message is set and there is a single flow and now branches in our code. The method form.show_message is called a single time on a single place in our code. If we change the API of this method we would change in a single place and will not forget about the second place. There is always a single scenario. Save->Show message.




     
  • kmitov 7:06 am on January 22, 2019 Permalink |
    Tags: cache, ,   

    Implementation of Multi-levels caching. Example for Rails 

    There are two difficult things in Computer Science. Choosing a name for a variable and cache invalidation.

    That being said I went on a journey to implement multi-levels caching in one of our platforms. Cache should be fast. Fast cache is expensive. If you can use 1M of very fast and expensive cache, why not implement a second level cache that is 10M, not that fast and not that expensive and 100M of normal cache that is cheap, but still faster than going to db.

    TL;DR;

    I decided to implement a DbCache that will store cached html rendered values directly in db and will access them from the DB instead of the fast Redis/Memcachier cache. All in the name of saving a few dollars on expensive fast cache that I do not really need.

    <% cache(cache_key) do # this is the redis cache %>
      <%= db_cache(cache_key) do # this is the db cache %>
        <% # actual calculation of value %>
      <% end %>
    <% end %>

    Implementation

    There is no need to constantly render the html/json of the content that we would like to serve to the client. It could be rendered once and served until updated. We are using Memcachier for a very fast access to cached values, but it is costing us more money. And in many cases we do not need this fast access.

    That’s why there is a DbCache implementation

    It works in the following way. It has a key and a value.

    When using in the view you can use

     <% cache(cache_key) do %>
       <%= db_cache(cache_key) do %>
       <% end %>
     <% end %>

    In this way if something is in cache we take it. This is L1 cache if you like. If it is not in L1 cache (that is stored in memory) than we ask db_cache. db_cache is our second level cache – L2. If the value is not in db_cache then we render it. The principle could be applied for L3 cache, although we are not there yet as a platform to need L3 cache.

    But it is db_cache. It is accessing the db. Why do you even call it cache?

    When the db_cache is accessed we make a single query to the db and retrieve a single record. For an indexed, not very large table this is fast. If the value is to be rendered again it will mean making a few more request for different objects and their associations and moving through all the trouble of rendering it again which involves views. By retrieving the HTML/JSON directly from DB we could directly serve it.

    How is db_cache implemented?

    DbCache model that stores the values in the db. It has a key and value columns. That’s it. Creating/retrieving/updating DbCache records is what is interesting.

    The key is column in the DB that is an integer. NOT a string. This integer is generated with a hash function and is than shifted right. The postgres db has a signed int column and the hash is generating an unsigned int. We have to shift because there is not enough space for storing unsigned int in a postgres db. In this way the cache key given from the developer is transformed to an internal key that is used for finding the record. And there of course is an index on this hash64 column.

    def db_cache key, &block
      # This will give us a 64 bit hash
      # Shift the value to reduce it because the column is signed and there is no room for
      # an unsigned value
      internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
      db_cache = DbCache.find_by(hash64: internal_key)
    
      ...
    end

    How are keys expired?

    If a key has expired we must create a new record for this key. Expiring keys could be difficult. Every record has an updated_at value. Every day a job on the db is run and if the updated_at value is more than specific days old it is automatically expired. This controls the number of records in the DB. I am kind of interested in storing only records that are regularly accessed. I think that if a page was not accessed in a couple of days, you generally do not need a cached value for it.

    This opens the next question:

    How are keys marked not to expire? If we change the accessed_at for a record on every read that will be slow because of a write to accessed_at

    True. It is important to expire old keys, but it is also important not to touch records on every read request because this will be very slow. If we make a touch on every request to the cache this will involve an update that will slow down the method. So an update is happening only once a day. See the db_cache.touch call below. The record could be accessed thousands of times today but there will be only one write to update the updated_at value. To touch the record.

    def db_cache key, &block
    
      internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
      db_cache = DbCache.find_by(hash64: internal_key)
      if db_cache.nil?
        # create cache value
      else
        db_cache.touch if db_cache.updated_at < Time.now - 1.day
      end
    
      ...
    end

    How fast is DbCache?

    These are just referenced values and of course this depends on the size of your cached values and the db and the load. In our specific case on Heroku we’ve found that the DbCache generally retrieves values in the range of 2 to 8 ms. In comparison the very fast Memcachier does this in the range of 2 to 8 ms.

    We also used NewRelic to look at the performance that end users are experiencing. And there was a large improvement because we could cache hundreds of MB of records in DB compared to the few MB for Memcachier that we are paying for.

    Rails specific details

    Since this code has to live in our platform and it is also bound to use some other rails object there are a few things more that I’ve done. Here is the full code that I hope gives a complete picture.

    # Author::    Kiril Mitov  
    # Copyright:: Copyright (c) 2018 Robopartans Group
    # License::   MIT
    module DbCacheHelper
    
      def db_cache key, &block
        # puts "db_cache: key: #{key}"
        result = nil
        if controller.respond_to?(:perform_caching) && controller.perform_caching
    
          # This will give us a 64 bit hash
          # Shift the value to reduce it because the column is signed and there is now room for
          # un unsigned value
          internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
          db_cache = DbCache.find_by(hash64: internal_key)
          if db_cache.nil?
            # puts "DBCache Miss: #{key}, #{internal_key}"
            Rails.logger.info "DBCache Miss: #{key}, #{internal_key}"
    
            content = capture(&block)
    
            # Use a rescue. This will make sure that if
            # a race condition occurs between the check for
            # existence of the db_cache and the actuall write
            # we will still be able to find the key.
            # This happens when two or more people access the site at exactly the
            # same time.
            begin
              # puts "DBCache: Trying to create"
              # puts "DBCache Count before find or create: #{DbCache.count}"
              db_cache = DbCache.find_or_create_by(hash64: internal_key)
              # puts "DBCache Count after find or create: #{DbCache.count}"
              # puts "DBCache: Found or record is with id:#{db_cache.id}"
            rescue ActiveRecord::RecordNotUnique
              retry
            end
    
            # The update is after the create because the value should not be part of the
            # create.
            db_cache.update(value: content)
          else
            # puts "DBCache Hit: #{key}, #{internal_key}"
            Rails.logger.info "DBCache Hit: #{key}, #{internal_key}"
            db_cache.touch if db_cache.updated_at < Time.now - 1.day
          end
    
          result = db_cache.value
        else
          result = capture(&block)
        end
        # Result could be nil if we've cached nil. So just dont return nil,
        # but return empty string
        result ? result.html_safe : ""
      end
    
    end


     
  • kmitov 2:02 pm on January 9, 2019 Permalink |
    Tags: i18n, parallel, , ,   

    i18n locales and the pass of rspec parallel specs and 

    First rule of good unit testing is: each test should be independent of the other tests.

    But if there is a global variable like I18n.locale than one spec could touch it and another spec will be run in a different locale from the default.

    TL;DR;

    Before each suite of specs set the locale to the default. This ensures that the specs are run against the same locale each time. Specific code is:

    # spec/rails_helper.rb
    RSpec.configure do |config|
      config.before :each do
        I18n.locale = Globalize.locale = I18n.default_locale
      end
    end
    

    i18n breaks spec isolation

    Internationalization, or i18n, should be part of most platforms. This means that i18n features should be properly tests. In suites when one of the specs modifies i18n the specs that are run after that are dependent on this new local.

    This seem trivial, but we only go to explore it the moment we started running specs in parallel on different CPU cores. The specs were started and run in different times and order on each run.

     
  • kmitov 7:21 am on January 8, 2019 Permalink |
    Tags: capybara, chromedriver, feature tests, google-chrome, , , ,   

    Chromedriver not filling all the whole password field in automated RSpec, Capybara, Feature Tests 

    This is such a lovely story. You will like it.

    When using google chromedriver to run capybara tests sometimes, just sometimes, especially if the tests are run in parallel, when the test has to fill a text field like a password, it fills only part of it. Last time checked for chromedriver 2.45

    TL; DR;

    Solution – google does not care, or it seems it is too difficult for the chromedriver team to resolve so there simply is no solution.

    What is the test?

    We are using Rails, Rspec, Capybara, Google chromedriver. We are developing feature tests. Tests are run in parallel with

    rake parallel:spec

    Here is the test in question. Simply fill a password on the form for upgrading a subscription, click on Confirm and expect to be redirected to a page that says – “You’ve upgraded your subscription”


    def submit_and_expect_success password
          # dialog opens to confirm with password
          fill_in "owner_password", with: password
          click_on "Confirm"
    
          expect_redirect_to "/subscriptions/#{subscription.to_param}"
    
          # If it redirects to this page, it means that the change was successful
          expect(page).to have_current_path "/subscriptions/#{subscription.to_param}"
    end

    And the tests are failing with timeout at expect_redirect_to. No, expect_redirect_to is a custom method, because we are using ActionCable to wait for subscription upgrade to finish. Because of the payment service at the back this sometimes takes a while and we want to show a nice progress and we need a websocket. But that being said the method is nothing special.

    module ExpectRedirect
      def expect_redirect_to url
        # If the path doesn't change before the timeout passes,
        # the test will fail, because there will be no redirect
    
        puts "expect url: #{url}"
        begin
          Timeout.timeout(Capybara.default_max_wait_time) do
            sleep(0.1) until url == URI(page.current_url).path
            page.current_url
          end
        rescue Timeout::Error=>e
          puts "Current url is still: #{page.current_url}"
          puts page.body
          raise e
        end
      end
    end

    If we are redirected to the url withing Capybara.default_max_wait_time than everything is fine. If not, we are raising the Timeout::Error.

    Parallel execution

    For some reason the test in question fails only when we are doing a parallel execution. Or at least mostly when we are doing parallel execution of the tests. So we moved through some nice articles to revise our understanding of Timeout again and again.

    https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/

    But nevertheless the tests were failing with Timeout::Error on waiting for a redirect and in the html we could see the error returned by the server:

    <div><p>Invalid password</p></div>

    How come the password is Invalid

    No this took a while to debug and it seems rather mysterious but this is what we got:

    User password in DB is: 10124ddeaf1a69e3748e308508d916b6

    The server receives from the html form: 10124ddeaf1a69e3748e30850

    User password in DB is: 74c2a3e926420e1a30363423f121fc1e

    The server receives from the html from: 74c2a3e926420e1a3

    and so on and so on.

    Sometimes the difference is 8 symbols. Sometimes it is 2. Sometimes it is 16.

    It seems to be a client side issue

    Like. JavaScript. If there is an error this strange it has to be in the JavaScript. Right. There in the javascript we see:

    let form_data = form.serializeObject();
    this.perform('start_change', form_data);

    The form gets serialized. Probably it is not serialized correctly. Probably the values that we are sending are just not the values on the form. So I revised my knowledge on serializing objects in JavaScript with

    https://stackoverflow.com/questions/17488660/difference-between-serialize-and-serializeobject-jquery

    So far so good. But the serialization was not the problem. Here is what I did. I fixed all the passwords to be 32 symbols.

    let form_data = form.serializeObject();
     if(form_data["owner_password"].lenght != 32) {
            form_data["owner_password"] = "this was:" + form_data["owner_password"] + " which is less than 32 symbols"
          }
    this.perform('start_change', form_data);

    It it happened. The value of the password field was simply not 32 symbols long. It was not filled during the test.

    A little bit of search and we arrive at:

    https://github.com/teamcapybara/capybara/issues/1890

    and there in the bottom of the issue, there is the standard: “Not our problem resolution” with the link to:

    https://bugs.chromium.org/p/chromedriver/issues/detail?id=1771&q=sendkeys&sort=-id&colspec=ID%20Status%20Pri%20Owner%20Summary

    It seems that google chromedriver is not filling all the characters in the password field. It is doing it on random and is completely unpredictable.

    Issue still exists on:

    Issue still exist for
    
    Chrome: Version 71.0.3578.98 (Official Build) (64-bit)
    Chromedriver chromedriver --version
    ChromeDriver 2.45.615279 (12b89733300bd268cff3b78fc76cb8f3a7cc44e5)
    Linux kireto-laptop3 4.4.0-141-generic #167-Ubuntu x86_64 GNU/Linux
    Description:	Ubuntu 16.04.5 LTS
    Release:	16.04
    Codename:	xenial

    Today we would try Firefox driver.


     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel