Recent Updates Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 9:32 pm on January 11, 2021 Permalink |
    Tags: acts_as_paranoid, , globalize-rails, , ,   

    The problems with acts_as_paranoid and generally with Soft Delete of records in Rails 

    I am about to refactor and completely remove acts_as_paranoid from one of the models in our platform. There are certain issues that were piling up in the last few years and it is now already difficult to support it. As I am about to tell this to colleagues tomorrow I thought to first structure the summary in an article and directly send the article to the whole team.

    If you are thinking about using acts_as_paranoid and generally soft delete your records then this article could give you a good understanding of what to expect.

    Disclaimer: I am not trying to roast acts_as_paranoid here. I think it is a great gem that I’ve used successfully for years and it has helped me save people’s work where they accidentally delete something. It could be exactly what you need. For us it got too entangled and dependent on workarounds with other gems and I am planning to remove it.

    acts_as_paranoid VS Globalize – problem 1

    We have a model called Material. Material has a title. The title is managed by globalize as we are delivering the content in a number of languages.

    class Material < ApplicationRecord
      acts_as_paranoid
      translates :title
    end

    The problem is that Globalize knows nothing about acts_as_paranoid. You can delete a Material, and it should delete the translations, but when you try to recover the Material then there is an error because of how the translations are implemented and the order in which the translations and the Material are recovered. Which record should be recovered first? Details are at https://github.com/ActsAsParanoid/acts_as_paranoid/issues/166#issuecomment-650974373, but as a quote here

    Ok, then I think I see what happens: ActsAsParanoid tries to recover your translations but their validation fails because they are recovered before the main object. When you just call recover this means the records are just not saved and the code proceeds to save the main record. However, when you call recover!, the validation failure leads to the exception you see.

    In this case, it seems, the main record needs to be recovered first. I wonder if that should be made the default.

    I have work around this and I used it like this for an year

    module TranslatableSoftdeletable
      extend ActiveSupport::Concern
    
      included do
    
        translation_class.class_eval do
          acts_as_paranoid
        end
    
        before_recover :restore_translations
      
      end
    
      def restore_translations
        translations.with_deleted.each do |translation|
          translation.deleted_at = nil
          translation.save(validate: false)
        end
        self.translations.reload
      end
    
    end

    acts_as_paranoid VS Globalize – problem 2

    Let’s say that you delete a Material. What is the title of this deleted material?

    material = Material.find(1)
    material.destroy
    material.title == ?

    Remember that the Material has all of its translations for the title in a table that just got soft deleted. So the correct answer is “nil”. The title of the delete material is nil.

    material = Material.find(1)
    material.destroy
    material.title == nil # is true

    You can workaround this with

    material.translations.with_deleted.where(locale: I18n.locale).first.title

    But this is quite ugly.

    acts_as_paranoid VS cancancan

    The material could have authors. An author is the connection between the User and the Material. We are using cancancan for the controllers.

    In the controller for a specific author we show only the models that are for this author. We have the following rule:

    can [:access, :read, :update, :destroy], clazz, authors: { user_id: user.id }
    

    Here the problem is that you can only return non deleted records. If you would like to implement a Trash/Recycle Bin/Bin functionality for the Materials that you can not reuse the rule. The reason is the cancancan can not have can? and cannot? with an sql statement and the “authors: {user_id: user.id}” is an sql statement.

    What we could do is to use scopes

    # in the ability
    can [:access, :read, :update, :destroy], Material, Material.with_authors(user.id)
    
    # In the Material
    scope :with_authors, -> (user_ids) {
        joins(:authors).where(authors: {user_id: user_ids})
    }
    

    We move the logic for authorization from the ability to the model. I can live with that. It is not that outrageous. But it does not work, because the association with authors will return empty authors for the Material as the materials are also soft deleted.

    acts_as_paranoid vs cancancan – case 2

    When declaring load_and_authorize in the controller it will not fetch records that are already deleted. So you can not use cancancan to load in the controller a record that is deleted and you must load it yourself.

    class MaterialsTrashController < ApplicationController
    
      before_action only: [:show, :edit, :update, :destroy] do
        @content_picture = Material.only_deleted.find(params[:id])
      end
    
    
      load_and_authorize_resource :content_picture, class: "Material", parent: false
    
    end

    This was ugly, but as we had one trash controller it was kind of acceptable. But with a second one it got more difficult and deviated from the other controllers logic.

    acts_as_paranoid vs ContentPictures

    Every Material has many ContentPictures on our platform. There is a ContentPictureRef model. One of the pictures could be the thumbnail. Once you delete a material what is the thumbnail that we can show for this material?

    As the content_picture_refs relation as also soft deleted we should change the logic for returning the thumbnail. We change it from

    def thumbnail
        thumbnail_ref = self.content_picture_refs.where(content_picture_type:  :thumbnail).first
    end

    to

    def thumbnail
        thumbnail_ref = self.content_picture_refs.with_deleted.where(content_picture_type:  :thumbnail).first
    end

    I can live with this. Even relation that we have for the deleted record we must call with “with_deleted”. ContentPictures, Authors and other relations all should be changed for us to be able to see the deleted materials in a table that represents the Trash.

    acts_as_paranoid vs Active Record Callbacks

    There is another issue right at https://github.com/ActsAsParanoid/acts_as_paranoid/issues/168

    It took me hours to debug it a few months back. The order of before_destroy should not really matter.

    class Episode < ApplicationRecord
       acts_as_paranoid
       before_destroy do 
          puts self.authors.count
       end
       has_many :authors, dependent: :destroy
    
    class Episode < ApplicationRecord
       before_destroy do 
          puts self.authors.count
       end
       acts_as_paranoid 
       has_many :authors, dependent: :destroy

    acts_as_paranoid vs belongs_to

    By using acts_as_paranoid belongs_to should be:

    class Author < ApplicationRecord
       belongs_to :material, with_deleted: true
    end

    I could also live with this. At a certain stage it was easier to just add with_deleted: true to the belongs_to relations.

    acts_as_paranoid vs Shrine

    So the Material has a Shrine attachment stored on Amazon S3. What should happen when you delete the Material. Should you delete the file from S3? Of course not. If you delete the file you will not be able to recover the file.

    The solution was to modify the Shrine Uploader

    class Attacher < Shrine::Attacher
    
      def activerecord_after_destroy 
        # Just dont call super and keep the file
        # Sine objects at BuildIn3D & FLLCasts are softdeleted we want to 
        # handle the deletion logic in another way.
        true
      end
    
    end
    

    I was living with this for months and did not had many issues.

    What Pain and Problem are we addressing?

    The current problem that I can not find a workaround for is the MaterialsTrashController that is using CanCanCan. All the solutions would require for this controller to be different than the rest of the controllers and my biggest concern is that this will later result in issues. I would like to have a single place were we check if a User has access to the Material and whether they could read or update it or recover it. If we split the logic of the MaterialsController and the MaterialsTrashController we would end up with a hidden and difficult to maintain duplication.

    But was is the real problem that we want to solve?

    On our platform we have authors for the instructions and we work close with them. I imagine one particular author that I will call TM (Taekwondon Master). So TM uploads a material and from time to time he incidentally could delete a material. That’s it. When he deletes a Material it should not be deleted, but rather put in a trash. Then when he deletes it from the trash he must confirm with a blood sample. That’s it. I just want to stop TM from losing any Material by accident.

    The solution is pretty simple.

    In the MaterialsController just show all the materials that do not have a :deleted_at column set.

    In the MaterialsTrashController just show only the Materials with :delete_at controller.

    I can solve the whole problem with one simple filter that would take me like 1 minute to implement. We don’t need any of the problems above. They simply will not exist.

    That’s it. I will start with the Material and I will move through the other models as we are implementing the additional Trash controllers for the other models.

     
  • kmitov 9:14 am on May 7, 2021 Permalink |
    Tags: ,   

    “[DOM] Input elements should have autocomplete attributes” 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    This is one of the things that could make a platform better. Here is how the warning looks like in the browser console.

    More information at – https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/autocomplete

    The autocomplete attributes could allow browsers, extensions and other agents guess what the user should do on this page. It could make it easier for the user. For example an extension could suggest a new password in the field, or could understand to fill the name of the user in the “name” field.

    Additionally we don’t like warnings.

    To check out the behavior, if you have a password manager for example go to

    https://www.fllcasts.com/users/sign_in

    or

    https://www.buildin3d.com/users/sign_in

     
  • kmitov 8:39 am on May 7, 2021 Permalink |
    Tags: csrf, , security   

    [Rails] Implementing an API token for post requests 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    At the BuildIn3D platform we provide clients with API to send certain HTTP POST requests. Questions is – how do we authenticate them.

    Here is one of the authentication steps – we implemented our own build_token. When authenticity_token for CSRF is available we also use the authenticity_token. But it is not always available because the authenticity_token depends on the session and the session cookie. But there might not be a session and a cookie in some cases and yet we still need some authentication. Here is how we do it.

    Generate a Unique encrypted token on the server side

    The server generates a token based on pass params. This could be username or password or other info.

        def to_build_token
          len   = ActiveSupport::MessageEncryptor.key_len
          salt  = Rails.appplicaton.secret_build_token_salt
          key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
          crypt = ActiveSupport::MessageEncryptor.new(key)
          encrypted_data = crypt.encrypt_and_sign(self.build_id)
          Base64.encode64(encrypted_data)
        end

    This will return a new token that has encrypted the build_id.

    encrypted_data = crypt.encrypt_and_sign(self.build_id)
    # We could easily add more things to encrypt, like user, or some params or anything you need to get back as information from the token when it is later submitted

    Given this token we can pass this token to the client. The token could expire after some time.

    We would require the client to send us this token on every request from now on. In this way we know that the client has authenticated with our server.

    Decryption of the token

    What we are trying to extract is the build_id from the token. The token is encrypted so the user can not know the secret information that is the build_id.

    def self.build_id_from_token token
      len   = ActiveSupport::MessageEncryptor.key_len
      salt  = Rails.application.secret_salt_for_build_token
      key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
      crypt = ActiveSupport::MessageEncryptor.new(key)
      crypt.decrypt_and_verify(Base64.decode64(token))
    end

    Requiring the param in each post request

    When a post request is made we should check that the token is available and it was generated from our server. This is with:

      def create
          build_token = params.require("build_token")
          build_id_from_token = Record.build_id_from_token(build_token)
          .... # other logic that now has the buid_id token
      end

    The build token is one of the things we use with the IS at BuildIn3D and FLLCasts.

    Polar bear approves of our security.

     
  • kmitov 8:15 am on May 7, 2021 Permalink |
    Tags: ,   

    [Rails] Crawlers you don’t want to cause a notification for an error 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We are using exception_notification gem. It is great because it will send us a direct email when there is an error on the platform. This means we are not looking at the logs. We get the notification directly on the email.

    But some of the crawlers (like all of them) cause errors, mainly in the cross-origin domain. The JavaScript we are delivering is not allowed for their domain and this generates an error. One way to stop these errors is to ignore errors caused crawlers in the exception_notifcation configuration:

    Rails.application.config.middleware.use ExceptionNotification::Rack,
        ignore_crawlers: %w{Googlebot bingbot YandexBot MJ12bot facebookexternalhit SEMrushBot AhrefsBot},
        :email => {
          :email_prefix => ...,
          :sender_address => ...,
          :exception_recipients => ...
        }

     
  • kmitov 8:05 am on May 7, 2021 Permalink |
    Tags: ,   

    [Rails] Disabling Forgery Protection on API controllers 

    Forgery protection comes for free in Ruby on Rails and is described in the security guide – https://guides.rubyonrails.org/security.html#cross-site-request-forgery-csrf.

    You don’t want forgery protection on some API controllers. If the API controllers extend the ActionController::Base the forgery protection could be disabled in the following way. Here is an example for a controller accepting requests from Amazon AWS SNS.

    class SnsController < ActionController::Base
    
      # Disable it for this controller. 
      # If there is no session it is just null session
      protect_from_forgery with: :null_session
    
      http_basic_authenticate_with :name => ENV["SNS_USERNAME"], :password => ENV["SNS_PASSWORD"]
    
      ...
    end
    

    An even better approach would be not extending from ActionController::Base, but from ActionController::API. But then we would have to include the modules for HttpAuthentication which is a topic for another article.

     
  • kmitov 7:56 am on May 7, 2021 Permalink |
    Tags: ,   

    [Rails] Please use symbols for polymorphic route arguments 

    This error occurred today with our platform. Issue is at https://github.com/rails/rails/issues/42157

    The issue occurs because of a new security patch with with Rails tracked with CVE-2021-22885.

    You can no longer call polymorphic_path or other dynamic path helpers with strings, because if this strings are provided by the user this could result in unwanted router helper calls.

    # previous call
    polymorphic_path([article, "authors"])
    # should now be
    polymorphic_path([article, "authors".to_sym])
    # or better
    polymorphic_path([article, :authors])

    Migration effort

    A perspective on how serious it is to upgrade – tl;dr – it is not – about half an hour.

    All the calls in our platform for polymorphic_path

    $ git grep  "polymorphic_path" | wc -l
    321

    All the file that have calls to polymorphic_path

    $ git grep -l  "polymorphic_path" | wc -l
    143

    Numbers of files that I’ve changed – 13
    Time it took me to review all of them -16 minutes, from 18:24 to 18:40

    Now I am waiting for specs to pass.

    I guess it is not a big change and could be migrate in half an hour. Most of our calls were using symbols already and only about 15 calls from 321 were with strings. These are 4% of all the calls.

     
  • kmitov 5:03 am on April 14, 2021 Permalink |
    Tags: , ,   

    Rendering plain text from the server is not plain text in the browser 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    One thing that is good to know:

    If you render plain text from a web framework like Rails you will not have plain text in the browser. You will still have HTML in the browser.

    This bite us with one of the RSpec scenarios that we were running and I thought I should share with the team and the whole community.

    IRL

    The real life scenario is the following.

    In the Rails controller we do

    class DigestMessagesController < ApplicationController
        def show
            render plain: "Text"
        end
    end
    

    What we would expect to see in the browser is a simple “Text” text. If fact this is exactly what is returned by the server – you can see in the “Response” tab.

    you can see that the response from the server is just “Text”

    But if you now Inspect the content of the page in the browser you will see that there is HTML inside.

    
    <html>
     <head></head>
     <body>
     <pre style="word-wrap: break-word; white-space: pre-wrap;">Text</pre>
     </body>
    </html>
    

    This HTML was not returned by the framework. This HTML was added by the browser. This is not good because an RSpec system scenario with Capybara will pass with rack_test driver, but will fail with a Selenium driver. rack_test driver will not add this HTML while all the major browsers will add it.

    scenario 'for text' do
          visit "/digest_messages/#{digest_message.id}?preview=plain"
          expect(page.body.to_s).to include("Text")
          # This expect here will fail for Selenium and will pass for rack_test driver.
          expect(page.body.to_s).not_to include("html")
    end
    

    I hope this is helpful and could save you a couple of minutes debugging.

     
  • kmitov 6:28 pm on April 7, 2021 Permalink |
    Tags: , , , rails-ujs, ,   

    From a ticket to deploy in an hour. jQuery runs the scripts, DOM does not. 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    This is an article of a production incidents with Rails/Stimulus/Rails-ujs/DOM and jQuery.

    We made mistake that made us lose 31 registrations on the FLLCasts and a few more on the BuildIn3D platform. The stack involved includes Ruby on Rails with Rails-ujs, Stimulus, DOM and jQuery. I will enter into some details about how it occurred, why and how we resolved it in under an hour. I am writing this article mainly to share with my our team, but I am sure it could be useful for other teams.

    FLLCasts registration form as of April 2021

    The incident – people could not Sign Up on the platforms

    On the Sign up page there is a captach. Years ago we found our that the captcha helps us reduce invalid registartion.
    There is also one more thing on this platform – usernames. As the FLLCasts platforms is used by students and academies, one of the very useful features on the platform is the automatically generated username. Users type their names in the “Full name” field and a username is automatically generated for them. In this way there is a unique username for users because email is not unique – there could be many users with the same email.

    To generate a new username we send a request to the server and refresh form. We use Stimulus and rails-ujs for this.

    The is an “change->registrations#refreshForm” for the input field for the name. When the full name is changed the form is refreshed and a unique username is returned from the server.

    <%= user_form.fields_for :data, {parent_builder: user_form} do |data_form| %>
      <%= data_form.text_field :name_attribute,
        {
          icon_prepend: "icon-finance-067 u-line-icon-pro",
          autofocus: true,
          data: {
            registrations_target: "name",
            action: "change->registrations#refreshForm"
          },
          autocomplete: "off"
        }
      %>
    
      <%= data_form.email_field :email %>
    <% end %>

    Here is how it should work:

    How the form is supposed to work

    The incident was that when the form was refresh the captcha got lost and we were not showing it after that. This means that on the server we were checking the captcha, but there was not captcha on the client.

    The ticket

    A wild ticket appears from a user.

    Hi,
    
    We wanted to enroll our Kid for Personal B programme and we were trying to Sign Up to pay online , but we are unable to sign up using our details.
    We get an error message though we tried a few times.
    
    Please help to sort this out.
    
    Thanks
    
    Regards

    This is a really wild ticket. The user wants to buy and they can not sign up. 1,2,3 go.

    The problem

    We’ve made the following commit in the registration form

    --- a/app/views/devise/registrations/new.js.erb
    +++ b/app/views/devise/registrations/new.js.erb
    @@ -1 +1 @@
    -$("#<%= @form_id %>").replaceWith("<%= escape_javascript(render 'form') %>")
    \ No newline at end of file
    +document.getElementById("<%=form_id%>").outerHTML= '<%= escape_javascript(render "#{partial_name}") %>';
    \ No newline at end of file

    We are in the process of removing jQuery from the code. I saw this jQueyr call and decided to change it to a simple DOM call. It should be the same, right? No…

    jQuery executes scrips while document.outerHTML does not

    I knew this. But I did not consider it in this commit. What happens is that we receive the new form from the server and replace the form on the page with the new form received from the server that contains the generate username.

    But in this form (and this form only on the whole platform) there is a recaptach. This recaptcha is a JavaScript and when dynamically changing the DOM the JavaScript must be executed. Well, jQuery automatically executed this for us. document.outerHTML does not.

    Developing and RSpec system spec.

    First I deployed the fix to production. It took about 5 minutes.

    After that I developed the spec. This is how the spec looks like:

    scenario "shows recaptach when dynamically reloading the form", js: true  do
        with_recaptcha_enabled do
          visit_sign_up
    
          expect(page).to have_recaptcha
    
          name_field.set unique_name
          email_field.set "#{SecureRandom.hex(16)}@example.com"
    
          # This means we have reloaded with js because we've set the email field 
          # and this has change the focus and there was a fire event for the name_field.
          #
          # We are now waiting for the username to appear on the screen
          expect(page).to have_username_filled
    
          expect(page).to have_recaptcha
        end
      end

    We expect that there is a recaptcha before and after refreshing the form.

    What surprised us

    Looking at the data in the time frame this commit was on production – we’ve received 9 registrations on the platform. In the same previous period we’ve receive 40 registrations.

    This means that we’ve lost 31 registrations on the platform.

    What surprised us is that 9 people managed to register. The only way you could register on the platform during this period is to first enter your username. If you enter the username on your own we do not refresh the form as we don’t have to generate a username for you. This means that about 25% of the people start with their username and about 75% of the users at FLLCasts start the registration with their name or email.

    Good to know. Good to know.

    Conclusion

    Looking at the issue we could have prevented it in a number of ways. In a test environment it is difficult to have a recaptcha because there is no way to test that the recaptcha works. After all that is the whole purpose of a recaptcha – to prevent bots and a Selenium driver is exactly a bot. But it turned out we’ve missed it as a scenario in the specs. It is always a missing spec.

     
  • kmitov 11:45 am on April 2, 2021 Permalink |
    Tags: ,   

    The magic of class methods for a Ruby on Rails controller and their configurations 

    Have you used gems that provide class methods for Ruby on Rails controllers that extend these controllers?

    Like the ones below:

    class TutorialsController < ApplicationController
      
      # This is from cancancan
      # It allows to loading and authorizing
      load_and_authorize_resource
      ...
    
      # This is from 'has_scope'. 
      has_scope :pref_order, only: :index, default: "suggested"
      
    end
    

    Today’s article is about how to implement such a method for our own concern.

    My specific case is duplication. I would like to develop a DuplicateResourceConcern that extends the controller and allows users to duplicate a record when they create a new record. This is something we’ve been using at FLLCasts and BuildIn3D for many years, but today I revised the implementation, improved it and decided to share the knowledge in a short article.

    The goal is to be able to do:

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      enable_duplicate
      ...  
    end

    With this logic when the user:

    1. visits /tutorials/new?resource_to_dup=123
    2. The title field in the form for new tutorial is pre-filled the title of Tutorial 123.

    Declare DuplicateResourceConcern and enable_duplicate

    First we declare DuplicateResourceConcern

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        # call the duplication
      end
    
    end

    It is a concern with one method that is ‘dup’

    We use the concern in the TutorialsController

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      
      def new
      end  
    end

    The method ‘dup’ is a private method of the controller. It is called before the :new action. If a param called ‘resource_to_dup=X’ is passed we would like to duplicate the title of Tutorial X.

    Simple Tutorial duplication

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        original_tutorial = Tutorial.find_by_id(params.permit(:resource_to_dup)[:resource_to_dup])
        @tutorial = Tutorial.new(title: original_tutorial.title)
      end
    
    end

    This duplication will work for the Tutorials controller. But it will not work for ArticlesController, because we are doing “Tutorial.find” and we are assigning a @tutorial variable.

    Reusing the DuplicateResourceConcern in a different controller

    To reuse the DuplicateResoureConcern we must call Article.find_by_id when we are in an ArticlesController and Tutorial.find_by_id when we are in a TutorialsController.

    Here is how we could do this using ‘controller_name’, ‘.classify’ and ‘.constantize’. We can get the name of the records from the controller name.

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      private
      def dup
        # This will return Tutorial for TutorialsController
        # and will return Article for ArticlesController
        clazz = controller_name.classify.constantize
    
        # If the controller is called 'tutorials' the instance name will be 'tutorial'
        instance_name = controller_name.singularize
    
        original_instance = clazz.find_by_id(params.permit(:resource_to_dup)[:resource_to_dup])
        duplicated_instance = clazz.new(title: original_instance.title)
    
        # After this call we will be able to access @tutorial in the TutorialsController and @article in the ArticlesController
        instance_variable_set("@#{instance_name}", duplicated_instance)
      end
    
    end

    In this way the DuplicateResourceConcern is not dependent on the type of the object. It could work for Articles and for Tutorials. It is still dependent on the property ‘title’, but I will leave this for now.

    How to configure the duplication

    The name of the parameter in the url should be ‘resource_to_dup’. What if we want to modify this name on the controller level. For example to have a parameter
    resource_to_dup‘ for TutorialsController and
    resource_to_copy‘ for ArticlesController.

    What we want to do is pass an argument with the name of the parameter, but we would have to do a little more work.

    class TutorialsController < ApplicationController
      
      include DuplicateResourceConcern
      enable_duplicate param_name: "my_new_param_name"
      
      def new
      end  
    end

    We would use the class_method for the concerns.

    module DuplicateResourceConcern
      extend ActiveSupport::Concern
    
      included do
        before_action :dup, only: [:new]
      end
    
      def self.included(base)
        base.class_eval do
          extend ClassMethods
          # Declare a class-level attribute whose value is inheritable by subclasses. Subclasses can change their own value and it will not impact parent class.
          # In this way we can extend the class and still have separate configurations
          class_attribute :duplicate_resource_configuration, instance_writer: false
      
          self.duplicate_resource_configuration = {
            options: {
              param_name: "resource_to_dup",
            }
          }
        end
      end
    
      class_methods do 
        # The enable_duplicate method will be called on a class level.
        def enable_duplicate(*args) do
          options = args.extract_options!
          options.symbolize_keys!
    
          # We check that there are only parameters with this names.
          options.assert_valid_keys(:param_name)
    
          # Merge the default options with the options passed in the controller
          self.duplicate_resource_configuration[:options].merge!(options)
    
          # We mark that we've enabled the duplication
          self.duplicate_resource_configuration[:enabled] = true
        end
    
      end
    
    
      private
      def dup
        # This will return Tutorial for TutorialsController
        # and will return Article for ArticlesController
        clazz = controller_name.classify.constantize
    
        # If the controller is called 'tutorials' the instance name will be 'tutorial'
        instance_name = controller_name.singularize
    
        # Param name. We are no longer dependent on 
        # the param name being resource_to_dup. It could 
        # be a different name.
        the_param_name = self.duplicate_resource_configuration[:options][:param_name]
    
        original_instance = clazz.find_by_id(params.permit(the_param_name)[the_param_name])
        duplicated_instance = clazz.new(title: original_instance.title)
    
        # After this call we will be able to access @tutorial in the TutorialsController and @article in the ArticlesController
        instance_variable_set("@#{instance_name}", duplicated_instance)
      end
    
    end

    Extensibility

    This is the basic structure. We can now add more options if we want to and more parameters to the configurations.

    What is the system RSpec for duplication.

    It is simple.

    scenario "/tutorials/new?resource_to_dup can duplicate a resource" do
        tutorial.update(title: SecureRandom.hex(10))
    
        visit "/tutorials/new?resource_to_dup=#{tutorial.id}"
        expect(page).to have_text "Duplicate from #{tutorial.title}"
    
        click_on "Create Tutorial"
        expect(page).to have_text "Tutorial was successfully created."
    
        tutorials = Tutorial.where(title: tutorial.title)
        expect(tutorials.count).to eq 2
      end

     
  • kmitov 6:34 am on April 2, 2021 Permalink |
    Tags: , databases, , new relic, performance, ,   

    Resolving a performance issue in a Rails platform. cancancan, New relic, N+1 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    Today’s article is about a performance issue that we had in one of our Rails platforms and how we resolved it. I enter into details of the specific root cause and how New relic was helpful in identifying the issue. This article is mainly targeted for our team, but I hope it would be useful for the community to see a practical example of cancancan and N+1.

    The root cause

    Every time we show a lesson on the FLLCasts platform we show the tutorials and materials of that lesson. But if the user does not have access to a tutorial, because it is not published or it requires a teachers subscription we do not show it in the lesson. We filter it from the lesson.

    We are using cancancan and we’ve added code for checking if every tutorial in the course could be accessed – whether you have the right subscription. This was not a problem on its own. The major problem was that we were showing a navigation of the course on the right that contains links to all the lessons with all the tutorials:

    FLLCasts Course nagivation showing links to all the tutorials
    Show of links to all the tutorials in the whole course in the navigation

    To show the navigation we looped through all the tutorials in the whole course and checked if the user has access to them. It was an N+1. This seems to be working when a course contains < 100 tutorials. But once the courses grew and contained more tutorials the logic got slow.

    People started noticing. We had to increase the number of Heroku dynos from time to time. Especially when a lot of students join and use the platform.

    There were two problems

    1. We don’t need to show all the tutorials and building instructions in the navigation. We just need to show them for the current lesson.
    2. We need to make a singe query to the DB that would get only the tutorials in that lesson and only the tutorials the user has access to.

    What was the performance impact

    During load when a lot of students were using the platform the navigation/_course_section partial was taking 41% of the show lesson request average of 5 seconds. There were about 100 find queries for a single lesson.

    Table with how expensive it was to show the navigation for large courses

    It was taking about 10-15 seconds for the server to respond and sometimes requests were failing.

    Chart showing slow responses
    Responses were slow. > 15 second sometimes.

    On the chart below we see how the moment people start opening courses the time for showing lessons (course_sections) starts growing.

    Chart showing sharp increase
    Sharp increase in time the moment we make requests

    Practically the platform was unusable in this case. Everything stopped.

    Solutions

    Change the navigation to include only the current lesson

    We decided that we don’t need to show all the tutorials from the whole course in the navigation, only the tutorials for the current lessons. This reduced the load because instead of returning 140 tutorials we were returning 20.

    Improve the call to cancancan

    One thing that we’ve missed is a call to cancancan that is can? :show for each record. It has slipped in the years and we haven’t noticed.

    records.each do |record|
       if can? :show, record
         ...
       end
    end

    I refactored it to use cancancan accessible_by method.

    @content_refs = @course_section
            .content_refs
            .accessible_by(current_ability)
            .order(:position)

    and added two new :index rules in the ContentRefs ability. The benefit is that in this way cancancan makes a single query to the db.

    module Abilities
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user
          can :index, ContentRef, { archived: [nil,false], hidden: false}
          can :index, ContentRef, { archived: true, hidden: [true,false], course_section: { authors: { user_id: user.id }}}
          ...
          can :read, ContentRef, { archived: [nil,false], hidden: true ,course_section: {course: {required_access_for_hidden_refs: {weight: 0..user_max_plan_weight}}}}
          ...
        end
      end
    end

    The third rule is the deal breaker. With it we check if the user has a subscription plan that would allow them to access the content ref.

    With these three rules we allow everybody to see non archive and non hidden content_refs, we also allow authors of the course_section to see archived and hidden content_refs where they are the author of the course_section and we allow allow everybody to see hidden content_refs if they have the proper subscription plan.

    Results

    Number of timeouts reduced to from 180 to 6

    jenkins@elvis:/var/log/fllcasts$ grep "Rack::Timeout::RequestTimeoutException" access.log | wc -l
    6
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210320.gz | wc -l
    183
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210319.gz | wc -l
    164

    Increase in throughout put does not increase the required time

    It does not matter whether we have 1 or 40 rpm for this controller. The time stays the same. Next we can focus on how to make it even faster, but this will do until the platform grows x10

     
  • kmitov 6:44 am on March 30, 2021 Permalink |
    Tags: , , , , , , ,   

    Dead code and one more way it could hit you back – by being loaded! 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We can all agree that dead code should be removed. What we sometimes fail to see is how much it could cost to leave dead code with our product. Today’s article is about an example of a “dead code” and how it hit us back 1.5 years after we stopped using it. It took us 3 hours to debug and find the root cause. The conclusion is that we should have done ‘git rm’ an year ago. We did not and the cost was 3 hours.

    This is a real life example article from our code base and it is design to share experience with our team, but I hope the whole community could benefit.

    What happened

    1. Jenkins build failed

    First time we saw it we identified that it is not something serious and it kept failing for a few days until we had time to address it in the next Sprint.

    2. Teaspoon, Jasmine, Bundler, Rails Engines

    In the project there was the production code in Rails.root along with a dummy test code for starting JavaScript specs in Rails.root/test/dummy/. We are using Teaspoon with Jasmine in a Rails engines. Rails engines tend to create a test/dummy folder in which you could put the code for a test app in which you want your JavaScript specs to be executed. The product code is loaded in the test app and specs are executed.

    About 1.5 years ago I did some experiments with this test app and made a few configurations. The goal was to have the production code along with some test app code both loaded in the test environment. In this way Teaspoon and Jasmine would see both the production and the test app code as coming from “production”. I literally remembered what my idea was and what I was trying to achieve. At the end I decided not to use the test app code, but to have only the production code located in Rails.root be loaded in the test environment.

    But I forgot to remove the test app code from test/dummy/app/assets/javascripts/

    3. Teaspoon Error

    The following error occurred. Here “IS” is a JavaScript namespace part of the production code.

    jenkins@vpszap6s:~/jobs/is-core Build and Release/workspace/test/dummy$ xvfb-run -a bundle exec rake teaspoon --verbose
    Starting the Teaspoon server...
    Teaspoon running default suite at http://127.0.0.1:36051/teaspoon/default
    ReferenceError: IS is not defined
    
    

    4. The error and debugging process

    Not sure what the error was exactly I tried to identify the difference between my machine and the server machine where the tests was failing. This server machine is called “elvis”.

    1. Bundler version – my version was 2.2.11 and the elvis version was 2.2.14. Tried with upgrade, but it did not resolve the issue.
    2. Chrome driver – since the tests were executed on chrome I saw the chrome driver versions were different. Mine was 86. elvis was with 89. Synced them, the error was still occurring.
    3. Rake version – rake versions were the same
    4. Ruby version – the same
    5. Teaspoon and Jasmine versions – the same
    6. The OS is the same
    7. I could not find any difference between my env and the elvis env.

    Turns out the problem was in the loading order. elvis was loading test app code and then production code. My machine was loading production code and then test app code. I could not figure out why. The whole test app code was:

    // test/dummy/app/assets/javascripts/ext/dummy/dummy.js 
    IS.Dummy = {};

    The error that was occurring was:

    ReferenceError: IS is not defined

    5. Solution

    I was curious to continue debugging to see why the two machines were loading the code in a different order, but at the end decided against it. Just removed the dead code in test/dummy/app/assets/javascripts/ext/dummy/dummy.js which I only thought as dead, but it turned out it was affecting our project.

    6. Builds passing

    Finally the builds were passing.

    Conclusion

    Dead code my not be that dead after all. It might be loaded even if not used and loading order could differ. Better be safe and remove dead code at all. If we need it that much, we have it in the GIT repo history.

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel