Updates from April, 2021 Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 11:45 am on April 2, 2021 Permalink |
    Tags: ,   

    The magic of class methods for a Ruby on Rails controller and their configurations 

    Have you used gems that provide class methods for Ruby on Rails controllers that extend these controllers?

    Like the ones below:

    class TutorialsController < ApplicationController
      
      # This is from cancancan
      # It allows to loading and authorizing
      load_and_authorize_resource
      ...
    
      # This is from 'has_scope'. 
      has_scope :pref_order, only: :index, default: "suggested"
      
    end
    

    Today’s article is about how to implement such a method for our own concern.

    My specific case is duplication. I would like to develop a DuplicateResourceConcern that extends the controller and allows users to duplicate a record when they create a new record. This is something we’ve been using at FLLCasts and BuildIn3D for many years, but today I revised the implementation, improved it and decided to share the knowledge in a short article.

    The goal is to be able to do:

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      enable_duplicate
      ...  
    end

    With this logic when the user:

    1. visits /tutorials/new?resource_to_dup=123
    2. The title field in the form for new tutorial is pre-filled the title of Tutorial 123.

    Declare DuplicateResourceConcern and enable_duplicate

    First we declare DuplicateResourceConcern

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        # call the duplication
      end
    
    end

    It is a concern with one method that is ‘dup’

    We use the concern in the TutorialsController

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      
      def new
      end  
    end

    The method ‘dup’ is a private method of the controller. It is called before the :new action. If a param called ‘resource_to_dup=X’ is passed we would like to duplicate the title of Tutorial X.

    Simple Tutorial duplication

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        original_tutorial = Tutorial.find_by_id(params.permit(:resource_to_dup)[:resource_to_dup])
        @tutorial = Tutorial.new(title: original_tutorial.title)
      end
    
    end

    This duplication will work for the Tutorials controller. But it will not work for ArticlesController, because we are doing “Tutorial.find” and we are assigning a @tutorial variable.

    Reusing the DuplicateResourceConcern in a different controller

    To reuse the DuplicateResoureConcern we must call Article.find_by_id when we are in an ArticlesController and Tutorial.find_by_id when we are in a TutorialsController.

    Here is how we could do this using ‘controller_name’, ‘.classify’ and ‘.constantize’. We can get the name of the records from the controller name.

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        # This will return Tutorial for TutorialsController
        # and will return Article for ArticlesController
        clazz = controller_name.classify.constantize
    
        # If the controller is called 'tutorials' the instance name will be 'tutorial'
        instance_name = controller_name.singularize
    
        original_instance = clazz.find_by_id(params.permit(:resource_to_dup)[:resource_to_dup])
        duplicated_instance = clazz.new(title: original_instance.title)
    
        # After this call we will be able to access @tutorial in the TutorialsController and @article in the ArticlesController
        instance_variable_set("@#{instance_name}", duplicated_instance)
      end
    
    end

    In this way the DuplicateResourceConcern is not dependent on the type of the object. It could work for Articles and for Tutorials. It is still dependent on the property ‘title’, but I will leave this for now.

    How to configure the duplication

    The name of the parameter in the url should be ‘resource_to_dup’. What if we want to modify this name on the controller level. For example to have a parameter
    resource_to_dup‘ for TutorialsController and
    resource_to_copy‘ for ArticlesController.

    What we want to do is pass an argument with the name of the parameter, but we would have to do a little more work.

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      enable_duplicate param_name: "my_new_param_name"
      
      def new
      end  
    end

    We would use the class_method for the concerns.

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      def self.included(base)
        base.class_eval do
          extend ClassMethods
          # Declare a class-level attribute whose value is inheritable by subclasses. Subclasses can change their own value and it will not impact parent class.
          # In this way we can extend the class and still have separate configurations
          class_attribute :duplicate_resource_configuration, instance_writer: false
      
          self.duplicate_resource_configuration = {
            options: {
              param_name: "resource_to_dup",
            }
          }
        end
      end
    
      class_methods do 
        # The enable_duplicate method will be called on a class level.
        def enable_duplicate(*args) do
          options = args.extract_options!
          options.symbolize_keys!
    
          # We check that there are only parameters with this names.
          options.assert_valid_keys(:param_name)
    
          # Merge the default options with the options passed in the controller
          self.duplicate_resource_configuration[:options].merge!(options)
    
          # We mark that we've enabled the duplication
          self.duplicate_resource_configuration[:enabled] = true
        end
    
      end
    
    
      private
      def dup
        # This will return Tutorial for TutorialsController
        # and will return Article for ArticlesController
        clazz = controller_name.classify.constantize
    
        # If the controller is called 'tutorials' the instance name will be 'tutorial'
        instance_name = controller_name.singularize
    
        # Param name. We are no longer dependent on 
        # the param name being resource_to_dup. It could 
        # be a different name.
        the_param_name = self.duplicate_resource_configuration[:options][:param_name]
    
        original_instance = clazz.find_by_id(params.permit(the_param_name)[the_param_name])
        duplicated_instance = clazz.new(title: original_instance.title)
    
        # After this call we will be able to access @tutorial in the TutorialsController and @article in the ArticlesController
        instance_variable_set("@#{instance_name}", duplicated_instance)
      end
    
    end

    Extensibility

    This is the basic structure. We can now add more options if we want to and more parameters to the configurations.

    What is the system RSpec for duplication.

    It is simple.

    scenario "/tutorials/new?resource_to_dup can duplicate a resource" do
        tutorial.update(title: SecureRandom.hex(10))
    
        visit "/tutorials/new?resource_to_dup=#{tutorial.id}"
        expect(page).to have_text "Duplicate from #{tutorial.title}"
    
        click_on "Create Tutorial"
        expect(page).to have_text "Tutorial was successfully created."
    
        tutorials = Tutorial.where(title: tutorial.title)
        expect(tutorials.count).to eq 2
      end

     
  • kmitov 6:34 am on April 2, 2021 Permalink |
    Tags: , databases, , new relic, performance, ,   

    Resolving a performance issue in a Rails platform. cancancan, New relic, N+1 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    Today’s article is about a performance issue that we had in one of our Rails platforms and how we resolved it. I enter into details of the specific root cause and how New relic was helpful in identifying the issue. This article is mainly targeted for our team, but I hope it would be useful for the community to see a practical example of cancancan and N+1.

    The root cause

    Every time we show a lesson on the FLLCasts platform we show the tutorials and materials of that lesson. But if the user does not have access to a tutorial, because it is not published or it requires a teachers subscription we do not show it in the lesson. We filter it from the lesson.

    We are using cancancan and we’ve added code for checking if every tutorial in the course could be accessed – whether you have the right subscription. This was not a problem on its own. The major problem was that we were showing a navigation of the course on the right that contains links to all the lessons with all the tutorials:

    FLLCasts Course nagivation showing links to all the tutorials
    Show of links to all the tutorials in the whole course in the navigation

    To show the navigation we looped through all the tutorials in the whole course and checked if the user has access to them. It was an N+1. This seems to be working when a course contains < 100 tutorials. But once the courses grew and contained more tutorials the logic got slow.

    People started noticing. We had to increase the number of Heroku dynos from time to time. Especially when a lot of students join and use the platform.

    There were two problems

    1. We don’t need to show all the tutorials and building instructions in the navigation. We just need to show them for the current lesson.
    2. We need to make a singe query to the DB that would get only the tutorials in that lesson and only the tutorials the user has access to.

    What was the performance impact

    During load when a lot of students were using the platform the navigation/_course_section partial was taking 41% of the show lesson request average of 5 seconds. There were about 100 find queries for a single lesson.

    Table with how expensive it was to show the navigation for large courses

    It was taking about 10-15 seconds for the server to respond and sometimes requests were failing.

    Chart showing slow responses
    Responses were slow. > 15 second sometimes.

    On the chart below we see how the moment people start opening courses the time for showing lessons (course_sections) starts growing.

    Chart showing sharp increase
    Sharp increase in time the moment we make requests

    Practically the platform was unusable in this case. Everything stopped.

    Solutions

    Change the navigation to include only the current lesson

    We decided that we don’t need to show all the tutorials from the whole course in the navigation, only the tutorials for the current lessons. This reduced the load because instead of returning 140 tutorials we were returning 20.

    Improve the call to cancancan

    One thing that we’ve missed is a call to cancancan that is can? :show for each record. It has slipped in the years and we haven’t noticed.

    records.each do |record|
       if can? :show, record
         ...
       end
    end

    I refactored it to use cancancan accessible_by method.

    @content_refs = @course_section
            .content_refs
            .accessible_by(current_ability)
            .order(:position)

    and added two new :index rules in the ContentRefs ability. The benefit is that in this way cancancan makes a single query to the db.

    module Abilities
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user
          can :index, ContentRef, { archived: [nil,false], hidden: false}
          can :index, ContentRef, { archived: true, hidden: [true,false], course_section: { authors: { user_id: user.id }}}
          ...
          can :read, ContentRef, { archived: [nil,false], hidden: true ,course_section: {course: {required_access_for_hidden_refs: {weight: 0..user_max_plan_weight}}}}
          ...
        end
      end
    end

    The third rule is the deal breaker. With it we check if the user has a subscription plan that would allow them to access the content ref.

    With these three rules we allow everybody to see non archive and non hidden content_refs, we also allow authors of the course_section to see archived and hidden content_refs where they are the author of the course_section and we allow allow everybody to see hidden content_refs if they have the proper subscription plan.

    Results

    Number of timeouts reduced to from 180 to 6

    jenkins@elvis:/var/log/fllcasts$ grep "Rack::Timeout::RequestTimeoutException" access.log | wc -l
    6
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210320.gz | wc -l
    183
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210319.gz | wc -l
    164

    Increase in throughout put does not increase the required time

    It does not matter whether we have 1 or 40 rpm for this controller. The time stays the same. Next we can focus on how to make it even faster, but this will do until the platform grows x10

     
  • kmitov 6:44 am on March 30, 2021 Permalink |
    Tags: , , , , , , ,   

    Dead code and one more way it could hit you back – by being loaded! 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We can all agree that dead code should be removed. What we sometimes fail to see is how much it could cost to leave dead code with our product. Today’s article is about an example of a “dead code” and how it hit us back 1.5 years after we stopped using it. It took us 3 hours to debug and find the root cause. The conclusion is that we should have done ‘git rm’ an year ago. We did not and the cost was 3 hours.

    This is a real life example article from our code base and it is design to share experience with our team, but I hope the whole community could benefit.

    What happened

    1. Jenkins build failed

    First time we saw it we identified that it is not something serious and it kept failing for a few days until we had time to address it in the next Sprint.

    2. Teaspoon, Jasmine, Bundler, Rails Engines

    In the project there was the production code in Rails.root along with a dummy test code for starting JavaScript specs in Rails.root/test/dummy/. We are using Teaspoon with Jasmine in a Rails engines. Rails engines tend to create a test/dummy folder in which you could put the code for a test app in which you want your JavaScript specs to be executed. The product code is loaded in the test app and specs are executed.

    About 1.5 years ago I did some experiments with this test app and made a few configurations. The goal was to have the production code along with some test app code both loaded in the test environment. In this way Teaspoon and Jasmine would see both the production and the test app code as coming from “production”. I literally remembered what my idea was and what I was trying to achieve. At the end I decided not to use the test app code, but to have only the production code located in Rails.root be loaded in the test environment.

    But I forgot to remove the test app code from test/dummy/app/assets/javascripts/

    3. Teaspoon Error

    The following error occurred. Here “IS” is a JavaScript namespace part of the production code.

    jenkins@vpszap6s:~/jobs/is-core Build and Release/workspace/test/dummy$ xvfb-run -a bundle exec rake teaspoon --verbose
    Starting the Teaspoon server...
    Teaspoon running default suite at http://127.0.0.1:36051/teaspoon/default
    ReferenceError: IS is not defined
    
    

    4. The error and debugging process

    Not sure what the error was exactly I tried to identify the difference between my machine and the server machine where the tests was failing. This server machine is called “elvis”.

    1. Bundler version – my version was 2.2.11 and the elvis version was 2.2.14. Tried with upgrade, but it did not resolve the issue.
    2. Chrome driver – since the tests were executed on chrome I saw the chrome driver versions were different. Mine was 86. elvis was with 89. Synced them, the error was still occurring.
    3. Rake version – rake versions were the same
    4. Ruby version – the same
    5. Teaspoon and Jasmine versions – the same
    6. The OS is the same
    7. I could not find any difference between my env and the elvis env.

    Turns out the problem was in the loading order. elvis was loading test app code and then production code. My machine was loading production code and then test app code. I could not figure out why. The whole test app code was:

    // test/dummy/app/assets/javascripts/ext/dummy/dummy.js 
    IS.Dummy = {};

    The error that was occurring was:

    ReferenceError: IS is not defined

    5. Solution

    I was curious to continue debugging to see why the two machines were loading the code in a different order, but at the end decided against it. Just removed the dead code in test/dummy/app/assets/javascripts/ext/dummy/dummy.js which I only thought as dead, but it turned out it was affecting our project.

    6. Builds passing

    Finally the builds were passing.

    Conclusion

    Dead code my not be that dead after all. It might be loaded even if not used and loading order could differ. Better be safe and remove dead code at all. If we need it that much, we have it in the GIT repo history.

     
  • kmitov 6:44 am on March 25, 2021 Permalink |
    Tags: , ,   

    A practical example for dependency injection 

    In today’s article I am sharing with our team an example I found while reading parts of the code in one of our platforms where we’ve created a form of ‘cyclic’ references with cancancan merges in Rails Controllers. I wondered for a while how we’ve managed to create it, how to avoid creating them in the future and I share this as an example with our team, but I hope the article could be useful for the whole community.

    I see this as a perfect example for the use of Dependency Injection.

    The example

    CourseSectionsAbilities is merged with ContentsAbility which is merged with ContentRefsAbilities which is then merged with CourseSectionAbilities. This means that CourseSectionsAbilities is merged twice and the second merge overrides the first merge.

    class CourseSectionsController < CommonController
    
      def current_ability
        # This is the first merge 
        # CoruseSectionsAbility is merged with ContentsAbility.
        @current_ability ||= Abilities::CourseSectionsAbility.new(current_user)
                               .merge(Abilities::ContentsAbility.new(current_user))
      end
    
    end
    
    module Abilities
      class ContentsAbility
        include CanCan::Ability
    
        def initialize user
          ...
          # This is the second merge
          # ContentsAbility is merged with ContentRefs ability
          merge ContentRefsAbility.new user
        end
      end
    end
    
    module Abilities
    
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user
          # This is the third merge
          # ContentRefsAbility is merged with CourseSectionsAbilily.
          # This overrides the first merge.
          merge CourseSectionsAbility.new(user)
          ...
        end
      end
    end

    How did it happen?

    It think it is the classic grow of the code where you solve the problem in the easiest way possible. Initially we had:

    class CourseSectionsController < CommonController
    
      def current_ability
        # This is the first merge 
        # CoruseSectionsAbility is merged with ContentsAbility.
        @current_ability ||= Abilities::CourseSectionsAbility.new(current_user)
                               .merge(Abilities::ContentsAbility.new(current_user))
      end
    
    end

    But then a new requirement about an year after that has come and there is a commit that adds the second merge:

    module Abilities
    
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user
          # This is the third merge
          # ContentRefsAbility is merged with CourseSectionsAbilily.
          # This overrides the first merge.
          merge CourseSectionsAbility.new(user)
          ...
        end
      end
    end

    What is the problem?

    The problem is that there is a class called ContentRefsAbility that can dependent on everything it wants. It can merge anything inside it with any consideration on what was already merged. It can set it’s own dependencies. This couples the ContentRefsAbility with the places it is used. Because we must take into consideration every place where ContentRefsAbility is used before changing it’s implementation.

    How Dependency Injection solves this?

    We pass the ability in the constructor

    module Abilities
    
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user, outside_ability
          # This is the third merge
          # ContentRefsAbility is merged with CourseSectionsAbilily.
          # This overrides the first merge.
          outside_ability.method1 # we call the method of the outside_ability
          ...
        end
      end
    end

    Instead of creating the ability in the class we pass the dependency from the outside. In this way we can control the dependency and choose different dependencies in different conditions.

    The ContentRefsAbility no longer depends on the specific implementation of outside ability, but it depends on the behavior we inject from the outside.

     
  • kmitov 7:52 am on March 24, 2021 Permalink |
    Tags: , , , , ,   

    We don’t need ‘therubyracer’ and the libv8 error for compiling native extensions 

    This article is about the error:

    Installing libv8 3.16.14.19 with native extensions
    Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    and the fact that we don’t need and probably you don’t need therubyracer.

    There are two solutions for the above error:

    # To install libv8 with a V8 without precompiling V8 
    $ gem install libv8 -v '3.16.14.19' -- --with-system-v8
    
    # Or to comment therubyracer in your Gemfile and not to use it.
    # gem 'therubyracer'

    We went for the later. Here are more details

    What is libv8 and therubyracer

    From libv8

    V8 is Google’s open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and in Node.js, among others.

    What therubyracer does is to embeds the V8 Javascript Interpreter into ruby.

    In Rails there is the execjs gem used in assets precompilations.

    ExecJS lets you run JavaScript code from Ruby. It automatically picks the best runtime available to evaluate your JavaScript program, then returns the result to you as a Ruby object.

    https://github.com/sstephenson/execjs

    This means that execjs needs a JS runtime and it uses the one provided by therubyracer, libv8 and V8. But it can use other runtimes as well, like Node.

    What is the libv8 compilation error?

    The libv8 error is because it can not compile V8 on the machine. V8 is developed in C++. To successfully compile it the machine must have all the dependencies needed for compilation. You can try to compile it from scratch or you could use an already compiled one. Compiling from scratch takes time to resolve all the dependencies.

    But we didn’t need therubyracer

    Turns out that in this specific platform we don’t need therubyracer, because we already have Node because of webpacker/webpack and we have it on all the machines and we have it on production on Heroku.

    therubyracer is also discouraged on Heroku –

    If you were previously using therubyracer or therubyracer-heroku, these gems are no longer required and strongly discouraged as these gems use a very large amount of memory.

    A version of Node is installed by the Ruby buildpack that will be used to compile your assets.

    https://devcenter.heroku.com/articles/rails-asset-pipeline#therubyracer

    therubyracer is great for starting quickly especially when there is no node. But considering there there was node on all of our machines we just don’t need therubyracer.

    As a result the slug size on heroku was reduced from 210M to 195.9M. No bad. I guess the memory consumption will also be reduced.

    remote: -----> Compressing...        
    remote:        Done: 210M        
    remote: -----> Launching...        
    remote:        Released v5616     
    
    remote: -----> Compressing...        
    remote:        Done: 195.9M        
    remote: -----> Launching...        
    remote:        Released v5617       

    How did we came about this error?

    Every Wednesday we do an automatic bundle update to see what’s new and update all the gems (of course if the tests pass). This Wednesday the build failed, because of the libv8 error and native compilation.

    It was just an unused dependency left in the Gemfile.

     
  • kmitov 10:20 am on February 4, 2021 Permalink |
    Tags: , ,   

    RSpec Matchers should be simple 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    Today on a review I saw an RSpec Matcher that was more complex than it should be. We’ve discussed it internally. I think this article could be useful to our team and to the community as a whole.

    An RSpec Matcher should be simple. It should return just ‘true’ or ‘false’

    IRL example of a complex RSpec Matcher

    This is the real live example of an RSpec Matcher from todays review

      RSpec::Matchers.define :have_content_picture_type_set_to do |picture_ref, content_picture_type, content_object|
        match do |page|
          content_object_undersore = content_object.model_name.to_s.underscore
          content_object_undersore = "tutorial" if content_object_undersore == "episode"
          content_object_undersore = "courses/#{content_object.course.to_param}/section" if content_object_undersore == "course_section"
    
          picture_ref_row = page.find(:xpath, "//tr[.//img[@src='#{picture_ref.content_picture.image_url}']]")
          js_agnostic_click_table_action picture_ref.to_param, "/author/#{content_object_undersore}s/#{content_object.to_param}/pictures/#{picture_ref.to_param}/edit", "Edit Picture"
    
          page.select content_picture_type, from: "content_picture_ref[content_picture_type]"
    
          click_on_and_expect "Update Picture", "Picture successfully updated"
          expect(page).to have_picture_ref_with_content_picture_type picture_ref, content_picture_type
        end
        failure_message do |page|
          "expected page to have content_picture with the specified type, but was #{page.body}"
        end
      end

    There is a lot to unpack here.

    1. We do some calculations of paths with the whole content_object_underscore logic
    2. Then we find something on the page with the page.find method
    3. Then we call ‘js_agnostic_click_table_action‘ which is a complex method
    4. Then we select something from an HTML select
    5. Then we click a button with the click_on_and_expect
    6. At the end we expect something

    We should have only step 6. That’s the purpose of an RSpec Matcher. Return true or false if the page contains something or not.

    Why has this happened?

    Because RSpec.define :matcher is a function like any other ruby function. And as a ruby function you can do whatever you want with it. You can call anything you want in this function.

    Why should an RSpec Matcher be simple?

    That’s the convention for RSpec Matchers. Imagine the following spec:

        visit_pictures material
        expect(page).to have_picture_ref_with_content_picture_type material.content_picture_refs.first, "Thumbnail"
    

    With this spec you would expect visit a page and check if there is something on the page. But now have_pictuer_ref_with_content_picture_type makes calls, changes the db, modifies the current page and there is no way for you to understand it from reading the spec. Let alone reuse the Matcher in another situation.

    What feature is this about?

    The feature above is about showing a picture on the animated 3D models and building instructions that we show at BuildIn3D. Authors could choose which picture is the Thumbnail – example is this FabBrix Hen :).

    FabBRIX Farm Animals, Hen in 3D building instructions
     
  • kmitov 7:31 am on February 3, 2021 Permalink |
    Tags: ,   

    Rails after_action order is in reverse 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We’ve used Rails after_action for controllers twice in our platforms and products. It allows you to execute something after a rails controller action.

    Why? What is the IRL use case?

    Users upload files for a 3D model or building instructions. We want to call an external service to convert this 3D model or buildin3d instructions and we schedule a job. We want to ping this external service. So we make a network request in the controller.

    This is never a good idea. Generally you want to return fast from the call to your controller action. On Heroku you have a 15 seconds time limit to return and as a user experience it is good to return a result right the way and to do your job on the background and to report on the progress with some clever JS.

    But for this specific case we have reasons to try to ping the external service. It generally takes about 0.5-1s, it is not a bottleneck for the user experience, and it works well. Except for the cases when it does not work well.

    Today was such a day. Heroku dynos were trying to connect with the service, but it was taking longer than 15 seconds. I don’t know why. So i decided to revise the implementation.

    What is special about after_action order?

    It is the reverse of before_action.

    We want to call the external service in an after action. In this way it is outside of the logic of the controller method and if it fails, we handle the exception in the after_action. It is a good separation. The real code is:

    module IsBackend::JenkinsWorkoff extend ActiveSupport::Concern
      included do
        after_action do
          unless @jenkins_rebuild == nil || @jenkins_rebuild.running?
            begin
              JenkinsClientFactory.force_jobs_workoff timeout: 10
            rescue => e
              config.logger.info e.message
            end
          end
        end
      end
    end
    

    The issue is with the order of the after_action when they are more then one.

    Generally before_action are executed in the order they are defined.

      before_action do
        # will be first
      end
    
      before_action except: [:index, :destroy] do
        # will be second
      end

    But after_action are executed in the reverse order

      after_action do
        # will be SECOND
      end
    
      after_action except: [:index, :destroy] do
        # will be FIRST
      end

    Why? I will leave the PR to explain it best – https://github.com/rails/rails/issues/5464

    How do we use it?

    When building the buildin3d instruction below we’ve pinged the external service in the build process. Because of the after_action we can enjoy this FabBrix dog.

    FabBRIX Pets, Dog in 3D building instructions
     
  • kmitov 4:58 pm on January 25, 2021 Permalink |
    Tags: liquid,   

    How we use Liquid template language? (part 1) 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    In FLLCasts and BuildIn3D platforms we use three template languages. ERB (as it is all Rails), Liquid and Mustache. This article gives a brief overview of how we are using and extending Liquid in our platforms. The main purpose is to give an overview for newcomers to our team, but I hope the developers community could benefit from it as a whole.

    What is Liquid?

    Quoting directly from the site

    Liquid is an open-source template language created by Shopify and written in Ruby. It is the backbone of Shopify themes and is used to load dynamic content on storefronts.

    https://shopify.github.io/liquid/

    You create a template, you feed it values, it produces a result:

    @template = Liquid::Template.parse("hi {{name}}") # Parses and compiles the template
    @template.render('name' => 'tobi')                # => "hi tobi"

    Why Liquid?

    Short answer – I thought: “It is good enough for Shopify so It should be useful for us”

    With so many template engines it really is an interesting question why would we use Liquid.

    When I made the decision years ago I was facing the following problem.

    How to give admins the ability to create email templates where they write the id of an Episode. Before the email is sent, the template should evaluate what the title of this Episode is and fill it in the HTML.

    The problem with Episode titles was the following. An author creates an Episode and sets the title to “Line following with LEGO Mindstorms”. An admin prepares a message for all users that the new “Line following with LEGO Mindstorms” tutorial is released. While the admins prepare the email and include the title of the tutorial in the Email, the title of the tutorial is changed to “Line following with LEGO Mindstorms EV3” by the author. So we end up with two titles of the tutorial. One in the email, and one on the site. This happens with Pictures also.

    We are sending regular emails to subscribed users. As we are sending the emails I wanted to give the opportunity for admins to create a template of the email and then when the email is sent, the template is evaluated. The solution was to give admins the ability to create a “digest_message” in the platform and to fill this digest message with template values. Like for example:

    Hello, {{ user.name }},
    This is what we recommend you start with:
    {% episode = Episode.find(1) %}
    {{ episode.title }}
    {{ episode.destroy SHOULD NOT BE ALLOWED}}

    This was my goal. To get an instance of Episode, but without all the dangerous methods like “.destroy”. As admin I should still be able to call “episode.title”, but I should not be able to call “episode.destroy”.

    This was how we started with Liquid. This was the first time we needed.

    How do we use Liquid?

    The code above is close to Liquid, but it is not Liquid. You can not use Episode.find(1) in Liquid and this is good. I don’t want this method to be available for admins to write in the emails. Because they could also write other, more dangerous methods.

    Liquid gave us the solution – Liquid Drops

    Liquid::Drop

    The idea of Liquid::Drop is that you get a real object and you can call this object real methods while the template is evaluated. Here is our Episode drop

    module LiquidDrops
      class EpisodeDrop < Liquid::Drop
        
        def initialize episode
          @episode = episode
        end
    
        def title
          @episode.title.try(:html_safe)
        end
    
        def description
          @episode.description.try(:html_safe)
        end
    
        def published_at
          @episode.published_at.strftime('%d %B %Y')
        end
    
      end
    end
    

    Here is how we use this drop in a Liquid Template.

    {% assign episode = episodes.1189 %}
    {% assign title = episode.title %} 
    {% assign description = episode.description %}
    <article>
      <h3>
        {{ title }}
      </h3>
      <p> {{ description }}</p>
    </article>

    As you can see we get an instance of episode 1189 and we can then call this instance some specific methods. Like episode.title or episode.description. This is not the real ActiveRecord object for Episode. This is a new instance that is wrapping the ActiveRecord object and is delegating specific methods to it. Liquid would only call methods that are available in the Drop.

    The tricky part is to tell Liquid that it should connect ‘episodes.1189’ with the specific Drop class. We pass the drops as a context to template.render as:

    @drops['episodes'] = LiquidDrops::EpisodesDrop.new
    template = Liquid::Template.parse(template_content)
    result = template.render(@drops)

    In this way I’ve managed to solve our first challenge. Allow admins to write emails that can dynamically, when rendered, get the current name of the Episode without giving them access to more dangerous methods like Episode#destroy. I could have easily used ERB, but Liquid was such a good fit. It was designed exactly for this and Shopify were using it for their themes.

    Screenshot and life example

    If you register to FLLCasts (https://www.fllcasts.com) you will receive a couple of emails and all of them use Liquid.

    If you don’t want to subscribe here is a screenshot from a recently sent email for a new Course.

    Example from an email that was rendered with Liquid

    You see the picture, title and description. These are all evaluated and added to the HTML with the use of Liquid.

    Conclusion

    This concludes the first part of this articles. In the next parts I will take a look at how we build Filters and Tags and how and why do we use them.

     
  • kmitov 10:25 am on January 22, 2021 Permalink |
    Tags: , , Google Closure Compiler, , ,   

    Quality in an event-driven plugin based browser framework. 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We’ve designed, developed from scratch and are running an event-driven plugin based browser framework that we call Instructions Steps (IS). It helps us visualize 3D models and building instructions on the BuildIn3D and the FLLCasts platforms. Currently it consists of hundreds of extensions separated in about ~50 repos with 587 releases. We’ve figured out a way to keep the quality of the whole framework at a really good level with almost no bugs and errors.

    This article is about how we are doing. The main purpose is to give an overview for newcomers to our team, but I hope the developers community could benefit from it as a whole.

    The IS architecture (for context)

    I will enter into the details about the architecture of IS in another article. For this article it is enough to say that IS consists of a really small core – 804 lines of code and a lot of extensions.

    There are many extension that are extending the framework. Most of them are under 200 lines. The framework is highly decoupled and “everything is an extension”. Look at the 3D model and building instruction below. The “previous” button is an extension. The “next” button is an extension. You could have “parts list”, “bill of materials”, play animations, fit and rotate the camera. These are all extensions. I took this idea for the way we were building plugins for Eclipse (many years ago).

    Sphere from Geosmart GeoSphere, but this time in 3D

    What is the problem with quality and how do we keep delivering a quality product?

    Event-driven plugin based architectures have many advantages – like decoupling the plugins which makes them more maintainable. It forces you to have clear API boundaries which also makes them more maintainable. But there are a lot of questions and drawbacks compared to a nice Monolith app. What should we test? Should we test a specific extension, or the repo or the extension as it is working with all of its direct dependencies. What kind of specs should we develop? Should we have small unit spec that tests the extension in an isolation or we should put all the hundreds of extensions and test them all together. How do we make make these decisions?

    Here are the few simple rules that we try to follow.

    Compilation and type checking with Google Closure Compiler in ADVANCE mode

    We use vanilla JavaScript. No TypeScript. There are reasons for this. Nothing against TypeScript actually. We use Google Closure Compiler to compile each and every extension.

    Here is an example of a declaration of an “interface method”

    /**
     * Loads the given url and returns a Promise that when resolved will provide the caller with a {@link IS.StepsTree.StepData}.
     *
     * @export
     * @param  {string|File} file - url to the file or a DOM File object to be loaded
     * @return {Promise} Promise that when resolved will provide a {@link IS.StepsTree.StepData} which is the root step
     */
    IS.StepsTree.IProvider.prototype.getStepsTree = function(file) {};

    Looking at the code we have the jsdoc annotations like “@param”, “@return”, “@export”. These are annotations that GCC understands and will check. It will check if the param is of the given type, it will check if the returned value is of the given type. It will check if the classes that implement this interface actually implement it.

    Google Closure Compiler (GCC) will check if we are trying to access properties and methods that are not available.

    As a general rule of thumb – compilers are strict. If they understand your code, and compile it, then your code fulfills a bare minimum of requirements.

    GCC has helped us a lot. It takes some time to get used to it and to learn all the annotations and how to use them and how to develop SDK and libraries that are compiled, but it pays off. I’ve previously shared about our experience with GCC. Here is one lecture that I gave a few times – https://github.com/thebravoman/google-closure-compiler-presentation/blob/main/gcc_presentation.md

    Each extension is tested in isolation only with its direct dependencies available

    The navigation extensions are located in the repo “is-navigation”. When we test the functionality of the “Next” button we don’t expect to also have the “Fullscreen” or the “Animations” extensions available.

    Each extension is tested automatically in isolation, because each extension should work on its own given that it is the only extension that is installed (and the direct dependencies of course). Which makes sense. We are building a framework, a platform. When we have a framework, platform or even OS we should be able to install one extension, app, or program and they should be able to work on their own.

    For testing the extensions we use Jasmine and Teaspoon and I wrote an article about how and why we do it.

    All extensions are tested together in the ‘release_pack’

    What teams building platforms and frameworks quickly find out is that all the extensions and apps can work separately, but there are a lot of cases where if you put them all together and install them, things start to get more difficult. An example are all the different problems different OS have. One program is affecting another program in an unpredictable way.

    So we’ve build the is-release_pack. What it does is to put all the extensions together and to run a few basic tests on all of them.

    It contains 1-2 specs that check that each extension is working in the general case and probably one or two very specific cases. All the other specific cases are tested in the extensions, not in the “release_pack”. We push everything we can to the specs of the specific extension, but we have a few “integration” specs that are in the is-release_pack. And it is beautiful.

    The downside of integration specs

    There is one major downside with integration specs:

    All of us, developers, are lazy when it comes to really building it right. Once we build the feature and we see that it is working after a day of work there is little motivation in us to spend the next 3 days on building it right. It just feels so go to have it working that you commit and move on to the next thing.

    When there is a problem and an integration spec is failing most of the time it is easier to go and “ease” the integration spec. Change the expects a bit. Modify them. Even remove them.

    Other times when we have to develop a specific spec for the specific extension it feels easier to develop an integration spec instead of 20 specific specs in the repo for the extension.

    Sooner or later you end up in one of these situations:

    1. There are no integration specs or they contain expects and assertions that can not validate that your product is working correctly.
    2. There are a ton of integration specs that are constantly and randomly failing from time to time. The “integration” specs suite also takes forever to pass as there are now so many “integration specs”.

    Both of this situations are highly undesirable.

    Resolving the downside of integration specs

    One thing I learn writing business plans when applying for different VC funding is RACI.

    There are people Responsible for the job, people Accountable for the job, people that could be Consulted and people that should be kept Informed.

    So who is Accountable for the delivery of the IS framework and for the framework working correctly with all extensions in the user browser?

    Ideally it should be one person. In our case – it is Me.

    We are all responsible for the implementation. But in a team one should be held Accountable if something is not working and not right. One is Accountable for not checking. This person could change of course, but at any given moment there is someone “starting the engine of the car” as it exists the factory. You should start the engine and make sure the car works. You are accountable for checking it. You might not be responsible if it does not start, but you are accountable for checking.

    With the is-release_pack we resolved this for us.

    Only the Accountable (Me in our case) has access to the is-release_pack and its specs. Nobody else. You can not add integration specs, you can not remove, you can not even change on your own. The person that is accountable should do it. We keep the number of specs to a mininum – one basic scenario for each extension and when appropriate 1-2 (but no more) very specific scenarios for each extension. In the release_pack we prefer to has scenarios that involve more than one extension. In fact if there is a scenario that involves all the extensions we would probably use it.

    Integration specs are coupling the extensions?

    Yes. They are. When one extension fails the integration spec for all the extensions fail. That is true. With hundreds of extensions if every day a different extension “fails” then you will not have a successful run of the integration suite in years.

    But the customer “does not care”. The integration spec is the closes spec to the customer experience. The users never interacts with a single extension. They interact with all extensions.

    In the same time if an extension has reached the release_pack and is failing the release pack , we will go and add a new spec, but not in the release_pack. We add it into the repo for the specific extension. This protects us from regressions.

    Conclusion

    By having a small subset of integration specs in a project to which only I have access and that is the final step in the release pipeline we’ve managed to stop hundreds of releases that would break existing clients, would lose a feature or introduce a bug.

    587 official releases already and it takes 5 to 10 minutes to release the whole framework. Integration specs are present in the release_pack, but we keep them to a minimum, each testing many extensions at once and making sure that a real life client scenario is working.

     
  • kmitov 7:54 am on January 21, 2021 Permalink |
    Tags: , , , , ,   

    How and why we test JavaScript – Jasmine and Teaspoon 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We are serious about specs. A good suite of specs has helped us many times especially when it comes to regressions. For the JavaScript part of our stack many of our specs and using Jasmine and ran with Teaspoon in the Browser environment. This article is about how we use Teaspoon and Jasmine to run our specs for pure JavaScript frameworks like Instructions Steps, and BABYLON.js. Both BuildIn3D and FLLCasts are powered by this specs, but we use them mostly for the Instructions Steps Framework.

    Why JavaScript specs?

    In the Rails community specs are important. In the JavaScript community – well, they are not that well respected. We can debate the reasons, but this is our observation.

    When we implement specs for the platforms we like to use Rails system specs. They give us exactly what we need. Let’s take a real life example.

    Filtering by category

    We visit the page at BuildIn3D for all the 3D models & building instructions. We filter them by brand. We click on a brand and see only the 3D models & instructions for this specific brand.

    Here is the spec. Short and sweet.

    scenario "/instructions shows the instructions and can be filted by category", js: true do
      # go to the third page as the materials are surely not there
      visit "/instructions?page=3" 
      expect(page).not_to have_link material1.title
    
      click_category category1
    
      # We make sure the url is correct
      expect(page).to have_current_path(/\/instructions\?in_categories%5B%5D=#{category1.id}$/, url: true)
    
      # We make sure that only material1 is shown and material2 is not shown
      # The materials are filtered by category
      expect(page).to have_link material1.title
      expect(page).not_to have_link material2.title
    endT

    The spec is a Rails system spec. Other articles enter into more details about them. The point is:

    With a Rails system spec we don’t concern ourselves with JavaScript.

    We visit the page, we click, we see that something has changed on the page – like new materials were shown and others were hidden.

    What’s the use case for JavaScript specs then?

    Take a look at the following embedded BuildIn3D instruction. It is coming live and it has a next button. Click on the next button.

    TRS the Turning Radio Satellite construction from GeoSmart and in 3D

    Here is the actual spec in JavaScript with Jasmine and ran everyday with Teaspoon.

    it("right arrow button triggers TriedMoveIterator with 1", function(done) {
        // This is a JavaScript spec that is inside the browser. It has access to all the APIs 
        // of the browser. 
        const eventRight = new KeyboardEvent("keydown", { key: "ArrowRight" });
        document.onkeydown(eventRight);
        
        // Wait for a specific event to occur in the Instructions Steps (IS) framework
        // We are inside the browser here. There is no communication with the server
    
        IS.EventsUtil.WaitFor(() => this.iteratorListener.event).then(() => {
          expect(this.iteratorListener.event.getSteps()).toEqual(1);
          done();
        });
      });
    

    This the how we use JavaScript and this is why we need them:

    We need JavaScript specs that are run inside the browser to communicate with other JavaScript objects and the browser APIs for JavaScript apps.

    In this specific case there is this “iteratorListener” that monitors how the user follows the instructions. We need access to it. Otherwise it gets quite difficult to test. We can not write a spec to test what are the pixels on the screen that are drawn after clicking next. This will be … difficult to put it mildly. We don’t have to do it. We need to know that clicking the next button has triggered the proper action which will then draw the actual Geometry and Colors on the screen.

    How we use Jasmine and Teaspoon to run the JavaScript specs

    Years ago I found a tool called Teaspoon. Looking back this has been one of the most useful tools we’ve used in our stack. It allows us to have a Rails project (and we have a lot of those) and to run JavaScript specs in this Rails project. The best thing – it just works (the Rails way).

    # add teaspoon dependency
    $ cd test/dummy/
    $ rails generate teaspoon:install
    # write a few specs
    $ rails s -p 8889
    

    You start a server, visit the url in the browser and the specs are executed

    Tests are pure JavaScript and we use Jasmine.

    That’s it. Not much to add. It’s simple The specs are a simple JS file located in “spec/javascripts/” folder and here is an example for one of them

    describe("IS.EventDef", function() {
      describe("constructing throws error if", function() {
        it("event name is null", function() {
          expect(() => new IS.EventDef(null, {})).toThrow(new Error("Null argument passed."));
        });
    
        it("event conf is null", function() {
          expect(() => new IS.EventDef("smoe", null)).toThrow(new Error("Null argument passed."));
        });
    
        it("declaring an event, but the class attribute value is null", function() {
          expect(() => {
            const ext = IS.ExtensionDef.CreateByExtensionConf({
              extension: new IS.Extension(),
              events: {
                declaredEvent: {
                  class: null
                }
              }
            });
          }).toThrow("Declared event 'declaredEvent' class attribute is null!");
        });
      });
    });
    

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel