Recent Updates Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 9:32 pm on January 11, 2021 Permalink |
    Tags: acts_as_paranoid, cancancan, globalize-rails, , ,   

    The problems with acts_as_paranoid and generally with Soft Delete of records in Rails 

    I am about to refactor and completely remove acts_as_paranoid from one of the models in our platform. There are certain issues that were piling up in the last few years and it is now already difficult to support it. As I am about to tell this to colleagues tomorrow I thought to first structure the summary in an article and directly send the article to the whole team.

    If you are thinking about using acts_as_paranoid and generally soft delete your records then this article could give you a good understanding of what to expect.

    Disclaimer: I am not trying to roast acts_as_paranoid here. I think it is a great gem that I’ve used successfully for years and it has helped me save people’s work where they accidentally delete something. It could be exactly what you need. For us it got too entangled and dependent on workarounds with other gems and I am planning to remove it.

    acts_as_paranoid VS Globalize – problem 1

    We have a model called Material. Material has a title. The title is managed by globalize as we are delivering the content in a number of languages.

    class Material < ApplicationRecord
      acts_as_paranoid
      translates :title
    end

    The problem is that Globalize knows nothing about acts_as_paranoid. You can delete a Material, and it should delete the translations, but when you try to recover the Material then there is an error because of how the translations are implemented and the order in which the translations and the Material are recovered. Which record should be recovered first? Details are at https://github.com/ActsAsParanoid/acts_as_paranoid/issues/166#issuecomment-650974373, but as a quote here

    Ok, then I think I see what happens: ActsAsParanoid tries to recover your translations but their validation fails because they are recovered before the main object. When you just call recover this means the records are just not saved and the code proceeds to save the main record. However, when you call recover!, the validation failure leads to the exception you see.

    In this case, it seems, the main record needs to be recovered first. I wonder if that should be made the default.

    I have work around this and I used it like this for an year

    module TranslatableSoftdeletable
      extend ActiveSupport::Concern
    
      included do
    
        translation_class.class_eval do
          acts_as_paranoid
        end
    
        before_recover :restore_translations
      
      end
    
      def restore_translations
        translations.with_deleted.each do |translation|
          translation.deleted_at = nil
          translation.save(validate: false)
        end
        self.translations.reload
      end
    
    end

    acts_as_paranoid VS Globalize – problem 2

    Let’s say that you delete a Material. What is the title of this deleted material?

    material = Material.find(1)
    material.destroy
    material.title == ?

    Remember that the Material has all of its translations for the title in a table that just got soft deleted. So the correct answer is “nil”. The title of the delete material is nil.

    material = Material.find(1)
    material.destroy
    material.title == nil # is true

    You can workaround this with

    material.translations.with_deleted.where(locale: I18n.locale).first.title

    But this is quite ugly.

    acts_as_paranoid VS cancancan

    The material could have authors. An author is the connection between the User and the Material. We are using cancancan for the controllers.

    In the controller for a specific author we show only the models that are for this author. We have the following rule:

    can [:access, :read, :update, :destroy], clazz, authors: { user_id: user.id }
    

    Here the problem is that you can only return non deleted records. If you would like to implement a Trash/Recycle Bin/Bin functionality for the Materials that you can not reuse the rule. The reason is the cancancan can not have can? and cannot? with an sql statement and the “authors: {user_id: user.id}” is an sql statement.

    What we could do is to use scopes

    # in the ability
    can [:access, :read, :update, :destroy], Material, Material.with_authors(user.id)
    
    # In the Material
    scope :with_authors, -> (user_ids) {
        joins(:authors).where(authors: {user_id: user_ids})
    }
    

    We move the logic for authorization from the ability to the model. I can live with that. It is not that outrageous. But it does not work, because the association with authors will return empty authors for the Material as the materials are also soft deleted.

    acts_as_paranoid vs cancancan – case 2

    When declaring load_and_authorize in the controller it will not fetch records that are already deleted. So you can not use cancancan to load in the controller a record that is deleted and you must load it yourself.

    class MaterialsTrashController < ApplicationController
    
      before_action only: [:show, :edit, :update, :destroy] do
        @content_picture = Material.only_deleted.find(params[:id])
      end
    
    
      load_and_authorize_resource :content_picture, class: "Material", parent: false
    
    end

    This was ugly, but as we had one trash controller it was kind of acceptable. But with a second one it got more difficult and deviated from the other controllers logic.

    acts_as_paranoid vs ContentPictures

    Every Material has many ContentPictures on our platform. There is a ContentPictureRef model. One of the pictures could be the thumbnail. Once you delete a material what is the thumbnail that we can show for this material?

    As the content_picture_refs relation as also soft deleted we should change the logic for returning the thumbnail. We change it from

    def thumbnail
        thumbnail_ref = self.content_picture_refs.where(content_picture_type:  :thumbnail).first
    end

    to

    def thumbnail
        thumbnail_ref = self.content_picture_refs.with_deleted.where(content_picture_type:  :thumbnail).first
    end

    I can live with this. Even relation that we have for the deleted record we must call with “with_deleted”. ContentPictures, Authors and other relations all should be changed for us to be able to see the deleted materials in a table that represents the Trash.

    acts_as_paranoid vs Active Record Callbacks

    There is another issue right at https://github.com/ActsAsParanoid/acts_as_paranoid/issues/168

    It took me hours to debug it a few months back. The order of before_destroy should not really matter.

    class Episode < ApplicationRecord
       acts_as_paranoid
       before_destroy do 
          puts self.authors.count
       end
       has_many :authors, dependent: :destroy
    
    class Episode < ApplicationRecord
       before_destroy do 
          puts self.authors.count
       end
       acts_as_paranoid 
       has_many :authors, dependent: :destroy

    acts_as_paranoid vs belongs_to

    By using acts_as_paranoid belongs_to should be:

    class Author < ApplicationRecord
       belongs_to :material, with_deleted: true
    end

    I could also live with this. At a certain stage it was easier to just add with_deleted: true to the belongs_to relations.

    acts_as_paranoid vs Shrine

    So the Material has a Shrine attachment stored on Amazon S3. What should happen when you delete the Material. Should you delete the file from S3? Of course not. If you delete the file you will not be able to recover the file.

    The solution was to modify the Shrine Uploader

    class Attacher < Shrine::Attacher
    
      def activerecord_after_destroy 
        # Just dont call super and keep the file
        # Sine objects at BuildIn3D & FLLCasts are softdeleted we want to 
        # handle the deletion logic in another way.
        true
      end
    
    end
    

    I was living with this for months and did not had many issues.

    What Pain and Problem are we addressing?

    The current problem that I can not find a workaround for is the MaterialsTrashController that is using CanCanCan. All the solutions would require for this controller to be different than the rest of the controllers and my biggest concern is that this will later result in issues. I would like to have a single place were we check if a User has access to the Material and whether they could read or update it or recover it. If we split the logic of the MaterialsController and the MaterialsTrashController we would end up with a hidden and difficult to maintain duplication.

    But was is the real problem that we want to solve?

    On our platform we have authors for the instructions and we work close with them. I imaging one particular author that I will call TM (Taekwondon Master). So TM uploads a material and from time to time he incidentally could delete and material. That’s it. When he deletes a Material it should not be deleted, but rather put in a trash. Then when he deletes it from the trash he must confirm with a blood sample. That’s he. I just want to stop TM from losing any Material by accident.

    The solution is pretty simple.

    In the MaterialsController just show all the materials that do not have a :deleted_at column set.

    In the MaterialsTrashController just show only the Materials with :delete_at controller.

    I can solve the whole problem with one simple filter that would take me like 1 minute to implement. We don’t need any of the problems above. They simply will not exist.

    That’s it. I will start with the Material and I will move through the other models as we are implementing the additional Trash controllers for the other models.

     
  • kmitov 10:15 am on December 16, 2020 Permalink |  

    FabBRIX demo 

    FabBRIX JUNGLE LIFE – Gorilla in 3D building instructions
     
  • kmitov 12:27 pm on December 10, 2020 Permalink |
    Tags: encoding   

    There are two ways to get the Cyrillic character ‘й’ in your string 

    [Everyday code]

    Today we had an issue with a Cyrillic character where some of the python scripts were not working. Turns out that the it is a know and special case. In summary, be careful when using Cyrillic character ‘й’ and pay special attention on how this files are processed.:

    In addition, you are probably hitting normalization issues. There are two ways to get the Cyrillic character 'й' in your string, one of them is a single code point, the other is two code points:>>> a = 'й'
    >>> b = 'й'
    >>> len(a), unicodedata.name(a)
    (1, 'CYRILLIC SMALL LETTER SHORT I')
    >>> len(b), unicodedata.name(b[0]), unicodedata.name(b[1])
    (2, 'CYRILLIC SMALL LETTER I', 'COMBINING BREVE')
     
  • kmitov 10:10 am on December 10, 2020 Permalink |
    Tags: , ,   

    Send array params to a rails server from a JavaScript code – URL and URLSearchParams 

    The goal was to filter building instructions on buildin3d.com by brand. We had to parse and create the URL on the client so I played around with URL and URLSearchParams (again). I will try to summarize the implementation here in the hope that you could help understand how URL and URLSearchParams work and how to use them with a Ruby on Rails app.

    Rails Server side request with array param

    The server accepts a request in the form:

    https://platform.buildin3d.com/instructions?in_categories[]=1&in_categories[]=2&in_categories[]=13

    This means we could pass an array with the ids of the categories. The result will return only the 3D assembly instructions that are for Brands in these categories.

    How to send the request on the client side

    On the client side we have an <ul> element with some <li> elements representing the categories.

    The brands filter is on the left.

    When we click on the brand we would like to add the brand to the URL and redirect the user to the new URL.

    So if the current url is

    https://platform.buildin3d.com/instructions?in_categories[]=1&in_categories[]=2

    and we select a new brand I would like to send the user to

    https://platform.buildin3d.com/instructions?in_categories[]=1&in_categories[]=2&in_categories[]=13

    How to add array params to the URL with JavaScript

    Here is the whole code of the Stimulus JS controller

    import { Controller } from "stimulus";
    
    /**
     * This is a controller used for filtering by brands [categories] on the materials index page
     *
     * @author Kiril Mitov
     */
    export default class extends Controller {
      static targets = ["tree"];
    
      connect() {
        console.log("connect");
        const scope = this;
        this.setFromUrl();
        this.treeTarget.addEventListener("click", e => {
          e.preventDefault();
          const li = e.target.closest("li");
          const input = li.querySelector("input");
          input.checked = !input.checked;
          scope.goToNewLocation();
        });
      }
    
      setFromUrl() {
        const url = new URL(window.location);
        const categoryIds = new URL(window.location).searchParams.getAll("in_categories[]")
    
        Array.from(this.treeTarget.querySelectorAll("li[data-category-id]"))
          .forEach(li => {
            const input = li.querySelector("input");
            const selected = categoryIds.indexOf(li.dataset["categoryId"]) != -1
            input.checked = selected;
          });
      }
    
      goToNewLocation() {
        const url = new URL(window.location)
        const searchParams = url.searchParams;
        searchParams.delete("in_categories[]");
        Array.from(this.treeTarget.querySelectorAll("li[data-category-id]"))
          .filter(li => li.querySelector("input").checked)
          .forEach(li => searchParams.append("in_categories[]",li.dataset["categoryId"]))
        searchParams.sort();
        window.location = url.toString();
      }
    }
    

    There are a few important things in the code

    Opening the page on a new location

    window.location = url.toString()

    This will open the new page for the user

    Adding the selected brands to the url search query

    We listen for an event of click from the user and we get a list of all the checked brands.

    goToNewLocation() {
        const url = new URL(window.location)
        const searchParams = url.searchParams;
        searchParams.delete("in_categories[]");
        Array.from(this.treeTarget.querySelectorAll("li[data-category-id]"))
          .filter(li => li.querySelector("input").checked)
          .forEach(li => searchParams.append("in_categories[]",li.dataset["categoryId"]))
        searchParams.sort();
        window.location = url.toString();
      }

    First we delete the param “in_categories[]”. We use URLSearchParams.delete. This removes the param from the searchParam and if we then call .toString() we would receive the new search query without this param.

    Then we call this.treeTarget.querySelectorAll to filter all the checkboxes and then only the check once and we append “in_categories[]” param for every selected checkbox. This is what the server requires.

    searchParams.append("in_categories[]",li.dataset["categoryId"])
    

    As a result with have the query

    https://platform.buildin3d.com/instructions?in_categories[]=1&in_categories[]=2

    Because we use only URLSearchParam.append and URLSearchParam.delete all the other params are still in the search query.

    As a summary:

    We have a Ruby on Rails server that accepts an array param and we form this array param in a Stimulus JS controller. We use URLSearchParams method to append and delete params, as this will preserve the other params that are already in the url.

     
  • kmitov 6:18 am on November 30, 2020 Permalink |
    Tags: , , ,   

    Testing an index page in a web application where we depend on the order. 

    [Everyday Code]

    Today the specs for an index page failed and I had to improve it. I decided to share a little trick for when we depend on the order of the objects.

    The specs is for the /subscriptions page where we show an index of all the subscriptions. We order the subscriptions by created_at in DESC. There are 20 subscriptions on the page. Then you must got to the next page. In the spec we open the page and check that there is a link to our subscription.

    visit "/admin/user/subscriptions"
    
    expect(page).to have_link subscription.random_id.to_s, href: "/admin/user/subscriptions/#{subscription.to_param}/edit"

    The problem was that there are other specs which for some reason create subscriptions into the future. This means that at a certain point in time when more than 20 subscriptions are created into the future in the DB, then our spec will fail. It will fail because the newly created subscription is on the second page.

    All the subscriptions on this page are into the future as today is 2020-11-30. So our newly created subscription is not here.

    What are our options?

    Move to the correct page in the spec

    This is an option. It will require us to have a loop in the spec that would more to the next page in the index and search for each subscription. This is too much logic for a spec.

    Delete all the future subscriptions before starting the spec

    Could be done. But is more logic for the spec. It actually needs to delete subscriptions and this is not the job of this specs.

    Create a subscription that is with the most recent created_at

    It is simple.

      let(:subscription) {FactoryBot.create(:subscription, created_at: subscription_created_at_time)}
    
      def subscription_created_at_time
        (Subscription.order(:created_at).last.try(:created_at) || Time.now)+1.second
      end
    
    

    It is the simpler change for this spec. Just create a subscription that’s last. In this way we know it will appear on the first page even when other specs have created subscriptions into the future.

     
  • kmitov 9:56 pm on November 28, 2020 Permalink |
    Tags: , ,   

    How to do headless specs with the BABYLON JS NullEngine 

    [Everyday code]

    In buildin3d.com we are using BABYLON JS. To develop a headless specs for BABYLON JS that could run in a Node.js environment or without the need of an actual canvas we can use BABYLON.NullEngine. A spec could then look like

     const engine = new BABYLON.NullEngine();
     this.scene = new BABYLON.Scene(engine);

    Here is what I found out.

    Headless specs

    We run a lot of specs for our BABYLON JS logic. All of this specs are against preview.babylonjs.com. The preview version of babylon give us access to the latest most recent changes of babylon that are mode available to the public. These are still not release changes, but are the work of progress of the framework. They are quite stable so I guess at least the internal suite of BABYLON has passed. Preview is much like a nightly build. In BABYLON JS case it is also quite stable.

    2 days ago much of our specs failed. I reported at https://forum.babylonjs.com/t/failure-error-typeerror-cannot-read-property-trackubosinframe-of-undefined/16087. There were a lot of errors for :

    Cannot read property 'trackUbosInFrame' of undefined.

    Turns out that many of our specs were using BABYLON.Engine to construct the scene like

    const engine = new BABYLON.Engine();
    this.scene = new BABYLON.Scene(engine);

    The BABYLON.Engine class is only intended for the cases where we have a canvas. What we should have been doing for headless specs without a canvas is to use BABYLON.NullEngine. Hope you find it helpful.

    Write more specs.

     
  • kmitov 9:39 pm on November 28, 2020 Permalink |
    Tags: , , , ,   

    Why we should never clear our DB before/after running specs. 

    One common “mistake” I’ve seen a couple of times is to clean the Database before/after specs are run. It seems to be a common practice with reasonable arguments. I think this is a bad idea. Here is why and what we should do instead.

    Why is the DB cleared before/after the specs

    When running specs that need access to a DB we might have to create a User or an Article or a Project model, then connect them in a certain way and test the business logic of our spec. After the spec is finished it is not wise to delete these objects from the DB directly in the spec. Sometimes it takes additional time, sometimes it executes additional logic. In most cases you don’t clear the DB after each and every spec.

    It is a good idea to clean the db before all the specs or after all the specs if they are successful. In this way we reset the DB only once, it saves some time and is much cleaner because you can plug in this behavior if you want to.

    Why the DB should not be cleared before/after the specs

    The simple answer is that our code will never, absolutely never work on a clean db in a production. If we have a test procedure that runs the specs against a clean and empty db they might pass when the db is clean. But what use do we have from code that could work in a clean environment, but could not work in a real production environment. The answer is – non.

    We don’t clean our db before/after each spec. In this way we’ve been able to track some really nasty bugs. Like slow queries that are slow only when you have too many users. Other cases involve special relations that are built in time. Like users that are part of an organization and the organization was once having one check for uniqueness of the user and now it has another check. Because the db is not cleared every time we make sure that it is properly migrated with all the needed migrations.

    We found out that a 7 years out test db that is not cleared is closer to a 7 years old production db.

    The test db is not the production db

    The test db is not the production db. It might have the same scheme, that is for sure, but the amount of data in them and the complexity of this data is different. What we need is code that could run on a production db. There is no use of any code that could run only in test environment.

    So here is what we do:

    We export the production db, we change some data like user emails, names and any other sensitive data and we import it as a test db. We run the specs on this db.

    In this way we make sure that the code could actually run on a real db before deploying it.

     
  • kmitov 5:36 pm on November 17, 2020 Permalink |
    Tags: , technical debt   

    Technical (code) Debt and how we handle it. 

    The subject of technical debt is interesting for me. I recently got a connection on linkedin offering me to help us identify, track and resolve technical debt and this compelled me to further write this article and to give some perspective on how we manage technical debt in our platforms and frameworks by specifically stopping on a few examples from the fllcasts.com and buildin3d.com platforms along with the Instructions Steps (IS) framework that we are developing. Hope it is useful for you all.

    What is technical debt?

    Here is the definition on the first source of wikipedia. It is pretty straightforward –

    Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution.

    (https://www.techopedia.com/definition/27913/technical-debt)

    What does it look like?

    The way I think about technical debt is – we write certain structures like for example “if” which in many cases is a Technical Debt. Of course there are many other types, but I will stop at this one for the article.

    // In this example we do a simple if for a step  when visualizing 3d assembly instructions
    // if there is animation for the specific step created by the author and persisted in the file we play this animation, but if there is no animation in the file we create a default animation.
    if(step.hasAnimation()) {
     return new AnimationFromFile(step)
    } else { 
     return new DefaultAnimation(step)
    }

    The problem with technical dept here is that there later in the development of the project two new cases might arise.

    Case 1 – we want to provide the functionality for AnimationFromFile(step) only to paying customers, otherwise we return just the default animation. The code then becomes:

    // We check if the user is subscribed and only then provide the better experience
    if(step.hasAnimation() && customer.hasSubscription()) {
     return new AnimationFromFile(step)
    } else { 
     return new DefaultAnimation(step)
    }

    Why is this bad? Where is the debt? – the debt is that we are now coupling the logic for playing animation with the logic for customers that have subscriptions. This means that when there are changes to the subscription logic and API we must also change the logic for handling animations.

    Case 2 – we introduce a third type of animation that is only for users with a certain WebGL feature in their browser. The code becomes:

    // We check if the browser supports the WebGL feature in question and then return a FancyAnimation.
    if(step.hasAnimation() && customer.hasSubscription()) {
     return new AnimationFromFile(step)
    } else {
     if (webGlFeaturePresent()) {
       return new FancyAnimation();
     else { 
       return new DefaultAnimation(step)
     }
    }

    Now we have a logic that knows about creating DefaultAnimation, about reading from files, about what a subscription is and when are users subscribed and it also knows much about the browsers and their support for WebGL.

    At a certain point in time we would have to refactor this logic and separate it into more decoupled pieces. That is a technical debt.

    How was Technical Debt created (in the example above)?

    We took the easy path now by placing one more if in the logic, but we knew that at some point we would have to refactor the logic.

    Should we omit Technical Debt?

    I think a good architecture could prevent a lot of the technical debt that is occurring. Good decoupled architecture with small units with clear boundaries and no state will result in 0 technical debt and we should strive to create such systems. Practically the world is not perfect. In a team of engineers even if you spend all your time on fighting with Technical Debt it is enough for only one colleague at one instance to take the easy path and add one more if to “fix this in 5 minutes instead of 3 hours” and the Technical Debt is already there. You’ve borrowed from the future.

    How do we track technical debt?

    I have personally learned to live with some technical debt. If I now do

    $ git grep "FIXME" 

    in one of our platforms we would get 37 results. These are 37 places where we think we could have made the implementation better and we even have some idea how, but we’ve actively decided that it is not the time now for this. Probably this part of the code is not used often enough. Or we are waiting for specific requirements from specific clients to come and we would address this them – when there is someone to “pay for it”. Can we address it now? – but of course. It would take us two, three days, but the question is why? Why should we address this now. Would it bring us more customers, would it bring more value to the customers? It would surely make our life easier. So it is a balance.

    Our balance

    I can summarize our balance like this.

    1. We identify part of the code as a technical Debt (because we do regular code reviews).
    2. We try to look at “what went wrong” and understand how this could be implemented better. We might even try in a different branch but we do not spend that much resources on this.
    3. We then know “what when wrong” and we agree to be more careful and not to take debt the next time but to instead implement it in the right way the first time.
    4. After that we decide if it is worth it to refactor the current issue – are the new clients coming that would ask us for modifications on these parts of the code?

    That’s it.

    Simple “FIXME”, “TODO”, “NOTE”, “IMPORTANT”, “SECURITY” tags in the code, git grep to see where we are and balance with trying to learn how to do it correctly next time.

    How can we solve Technical Debt for the example above?

    In buildin3d we have a framework with an event-driven plugin architecture. So for us it was simply a matter of registering a different plugin for the different features.

    // Pseudo code is 
    framework.register(new FancyAnimation())
    framework.register(new AnimationFromFile()) 
    framework.register(new DefaultAnimationExtension())
    
    framework.ariveOnStep((step)=> {
      ...ask all the extensions for animation and play the first animation that is returned
    }})

    The question is at which stage do you invest in a framework.

    What about MVP(s)?

    The greater balance is sometimes between an MVP and a working product. On one occasion we had an open source tool that was doing exactly what we needed. It was converting one 3D file format to a different 3D file format. We started the project. We used the open source tool. We delivered a working MVP in about a month and we took a lot of debt, because this tool came with other dependencies and was clearly not developed to be supported and extended. It was clear from the beginning that once new client requirements started coming we would have to re-write almost everything. And we waited. We waited for about 2 years. For 2 years we were extending the initial implementation and one day a client came with a requirement that we could no longer support. Then, it took us about 6 months to re-write the whole implementation in a completely new, much more extensible way and could easily accommodate new requirements.

    Conclusion

    Try not to take technical debt.

    If you have to then at least try to learn why it happens and learn how not to do it in the future. You will exponentially become better.

    Write down a comment in the code about why do you think this is a debt and how it should be approached. Spend some time reviewing and resolving debts if it pays off.

     
  • kmitov 11:05 am on November 16, 2020 Permalink |
    Tags: , animations, ,   

    We got featured in the BABYLON JS 4.2 release video 

    About two year ago when we started working on delivering 3D instructions and playing 3D animations and visualizing 3D models in the browser. Little did we know than that with version 4.2 babylon js will feature us in their release video. Thank you BABYLON theme.

    I wrote an article on BuildIn3D.com – https://buildin3d.com/blog/buildin3d-featured-in-the-babylon-js-4-2-release-video/

    Here is the release video for all to enjoy.

     
  • kmitov 6:48 am on November 2, 2020 Permalink |
    Tags: , ,   

    A week ago I gave a nice lecture about Google Closure Compiler and how to use it in ADVANCED_OPTIMIZATION mode. It is available in Bulgarian at https://softuni.bg/trainings/3194/advance-javascript-compilation-with-google-closure-compiler-why-and-how

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel