Tagged: fllcasts Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 9:01 am on April 6, 2022 Permalink |
    Tags: , fllcasts, ui, ux   

    How a bad UI message prevented us from resolving a 5 days feature downtime 

    This article is about a UI message in our platform that was confusing enough that it prevented us from resolving a 5 days downtime on a feature. We could have resolved it much faster, but because the message was confusing it made things worse. I hope that UI experts and engineers (like us) could benefit and get an insight of how some things are confusing to the users.

    Context

    Our https://www.buildin3d.com service for visualizing animated and interactive 3D assembly instructions is used by one of the well known robotics education platforms https://www.fllcasts.com. At FLLCasts there are instructions of LEGO robots and mission models from the FIRST LEGO League robotics competitions. Here is an example instruction – https://www.fllcasts.com/materials/1393-bag-12-helicopter-with-food-package-first-lego-league-2021-2022-cargo-connect

    On the left you see the message “Downloading initial steps…’

    Problem

    5 days ago a user wrote a ticket to us with a question:

    “Why can’t I download the instructions?”.


    I naturally responded to the user “Well, you can’t because this is how we’ve built them to work. They are open in the browser and they can not be downloaded. What are you trying to do anyway?” (oh, my ignorance)

    The user replied in the ticket

    “I just want to build them.”

    Obviously there was a great miscommunication between us. The user was trying to build the instructions, but was using the term “Download”. Why? Well, because we taught them so.

    5 days later I found out the following thing. Some of the instructions were not working. When users were visiting them we, as “ignorant engineers” were showing a message that said “Downloading initial steps…” while the instruction was loading. In our engineering mind we were really downloading the instruction to the browser. Some of the steps of the instruction were downloaded from the server to the client, which is the browser.

    When the user got in touch with us he said “Hey, why can’t I download the instructions”.
    When asked this question I assumed in the ticket that the user wanted to download the instructions on their file system and be able to open them from there in an offline mode. Which is generally what “Download” means. I made a really wrong assumption of what the user was trying to do. He was trying to load and view the instructions. He was not trying to “Download” them. But we were telling them that we are “Downloading” the instructions and they naturally used this term.

    The implications

    For 5 days about 5-6% of our instructions were not working correctly. We could have resolved this the right way if I paid a little more attention to what the user was asking in the ticket or if the message was “Loading initial steps…” instead of “Downloading initial steps…”

    You learn something every day.

     
  • kmitov 6:34 am on April 2, 2021 Permalink |
    Tags: , databases, fllcasts, new relic, performance, ,   

    Resolving a performance issue in a Rails platform. cancancan, New relic, N+1 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    Today’s article is about a performance issue that we had in one of our Rails platforms and how we resolved it. I enter into details of the specific root cause and how New relic was helpful in identifying the issue. This article is mainly targeted for our team, but I hope it would be useful for the community to see a practical example of cancancan and N+1.

    The root cause

    Every time we show a lesson on the FLLCasts platform we show the tutorials and materials of that lesson. But if the user does not have access to a tutorial, because it is not published or it requires a teachers subscription we do not show it in the lesson. We filter it from the lesson.

    We are using cancancan and we’ve added code for checking if every tutorial in the course could be accessed – whether you have the right subscription. This was not a problem on its own. The major problem was that we were showing a navigation of the course on the right that contains links to all the lessons with all the tutorials:

    FLLCasts Course nagivation showing links to all the tutorials
    Show of links to all the tutorials in the whole course in the navigation

    To show the navigation we looped through all the tutorials in the whole course and checked if the user has access to them. It was an N+1. This seems to be working when a course contains < 100 tutorials. But once the courses grew and contained more tutorials the logic got slow.

    People started noticing. We had to increase the number of Heroku dynos from time to time. Especially when a lot of students join and use the platform.

    There were two problems

    1. We don’t need to show all the tutorials and building instructions in the navigation. We just need to show them for the current lesson.
    2. We need to make a singe query to the DB that would get only the tutorials in that lesson and only the tutorials the user has access to.

    What was the performance impact

    During load when a lot of students were using the platform the navigation/_course_section partial was taking 41% of the show lesson request average of 5 seconds. There were about 100 find queries for a single lesson.

    Table with how expensive it was to show the navigation for large courses

    It was taking about 10-15 seconds for the server to respond and sometimes requests were failing.

    Chart showing slow responses
    Responses were slow. > 15 second sometimes.

    On the chart below we see how the moment people start opening courses the time for showing lessons (course_sections) starts growing.

    Chart showing sharp increase
    Sharp increase in time the moment we make requests

    Practically the platform was unusable in this case. Everything stopped.

    Solutions

    Change the navigation to include only the current lesson

    We decided that we don’t need to show all the tutorials from the whole course in the navigation, only the tutorials for the current lessons. This reduced the load because instead of returning 140 tutorials we were returning 20.

    Improve the call to cancancan

    One thing that we’ve missed is a call to cancancan that is can? :show for each record. It has slipped in the years and we haven’t noticed.

    records.each do |record|
       if can? :show, record
         ...
       end
    end

    I refactored it to use cancancan accessible_by method.

    @content_refs = @course_section
            .content_refs
            .accessible_by(current_ability)
            .order(:position)

    and added two new :index rules in the ContentRefs ability. The benefit is that in this way cancancan makes a single query to the db.

    module Abilities
      class ContentRefsAbility
        include CanCan::Ability
    
        def initialize user
          can :index, ContentRef, { archived: [nil,false], hidden: false}
          can :index, ContentRef, { archived: true, hidden: [true,false], course_section: { authors: { user_id: user.id }}}
          ...
          can :read, ContentRef, { archived: [nil,false], hidden: true ,course_section: {course: {required_access_for_hidden_refs: {weight: 0..user_max_plan_weight}}}}
          ...
        end
      end
    end

    The third rule is the deal breaker. With it we check if the user has a subscription plan that would allow them to access the content ref.

    With these three rules we allow everybody to see non archive and non hidden content_refs, we also allow authors of the course_section to see archived and hidden content_refs where they are the author of the course_section and we allow allow everybody to see hidden content_refs if they have the proper subscription plan.

    Results

    Number of timeouts reduced to from 180 to 6

    jenkins@elvis:/var/log/fllcasts$ grep "Rack::Timeout::RequestTimeoutException" access.log | wc -l
    6
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210320.gz | wc -l
    183
    jenkins@elvis:/var/log/fllcasts$ zgrep "Rack::Timeout::RequestTimeoutException" access.log-20210319.gz | wc -l
    164

    Increase in throughout put does not increase the required time

    It does not matter whether we have 1 or 40 rpm for this controller. The time stays the same. Next we can focus on how to make it even faster, but this will do until the platform grows x10

     
  • kmitov 8:31 pm on January 28, 2021 Permalink |
    Tags: , fllcasts, ,   

    How the software becomes unmaintainable? – a practical example 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    An ugly, sometimes over neglected truth of the industry is that the software that we work on does not become unmaintainable and tedious and difficult to work with overnight. It becomes like this with every change we introduce. While each individual change might not be “that bad” when they pile up we no longer have a well decoupled, maintainable system. We end up with a mess, which we than re-write with a new framework, a new team, new concepts and architecture just trying to do better. But the problem most of the time is in the single change that we do today – is it making the software system better or worse?

    This article is about a practical example of today and how we got to stop it.

    To better or worse

    When developing software incrementally we introduce changes. There are basically two options – the new changes would make the system better or they will make it worse.

    Here is the change from today

    --- a/file1.rb
    +++ b/file1.rb
    @@ -8,6 +8,7 @@ module IsBackend
     
         def embed
    +      @full_video_src = @material.video_refs.where(usage_type: "full_video").first.try(:video).try(:source_url)
     
    diff --git a/file2.rb b/file2.rb
    index bce7cd1a0..43b3768ea 100644
    --- a/file2.rb
    +++ b/file2.rb
     
       before_action do
         @namespaces = [:author]
    +    @full_video_src = @material.video_refs.where(usage_type: "full_video").first.try(:video).try(:source_url)
       end
    
    diff --git a/file3.rb b/file3.rb
    index 650456cb0..8c772288a 100644
    --- a/file3.rb
    +++ b/file3.rb
    @@ -41,6 +41,8 @@ class MaterialsController < CommonController
         @client_import_path = "shared" if Rails.application.config.platform.id == "b3"
    +    @preview_video_src = @material.video_refs.where(usage_type: "preview").first.try(:video).try(:source_url)
    +    @full_video_src = @material.video_refs.where(usage_type: "full_video").first.try(:video).try(:source_url)
       end
     

    The logic is not important. We basically get the video for a material. What’s important is the code is practically the same in all 3 places. 4 calls in three files and they are all the same.

    In the past I was leading a class in Software Development. One thing I tried to teach each student was:

    When you copy and paste you introduce a bug. That’s a fact of the industry.

    The reason is that once you copy/paste you would have to support the same logic in more than one place and you would simply forget about the second place the next time you would like to change the logic. It might not be you, it might be colleagues working years from now on the same code, but they will forget to change both places. That’s the problem with redundancy.

    Remove the redundancy, and you remove most of the bugs

    How did we stop it?

    Simple review. Just ask a second person to check your commit. This simple review process protects you from a large portion of the other bugs that are still left after you’ve cleared all the redundancies (of course).

    Enjoy

    The code in question is part of the logic delivering these instructions. See how the video is displayed. That’s what the feature was about. Enjoy.

    Torvi – Ball shooting Lego machine
     
  • kmitov 7:54 am on January 21, 2021 Permalink |
    Tags: , fllcasts, , , ,   

    How and why we test JavaScript – Jasmine and Teaspoon 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    We are serious about specs. A good suite of specs has helped us many times especially when it comes to regressions. For the JavaScript part of our stack many of our specs and using Jasmine and ran with Teaspoon in the Browser environment. This article is about how we use Teaspoon and Jasmine to run our specs for pure JavaScript frameworks like Instructions Steps, and BABYLON.js. Both BuildIn3D and FLLCasts are powered by this specs, but we use them mostly for the Instructions Steps Framework.

    Why JavaScript specs?

    In the Rails community specs are important. In the JavaScript community – well, they are not that well respected. We can debate the reasons, but this is our observation.

    When we implement specs for the platforms we like to use Rails system specs. They give us exactly what we need. Let’s take a real life example.

    Filtering by category

    We visit the page at BuildIn3D for all the 3D models & building instructions. We filter them by brand. We click on a brand and see only the 3D models & instructions for this specific brand.

    Here is the spec. Short and sweet.

    scenario "/instructions shows the instructions and can be filted by category", js: true do
      # go to the third page as the materials are surely not there
      visit "/instructions?page=3" 
      expect(page).not_to have_link material1.title
    
      click_category category1
    
      # We make sure the url is correct
      expect(page).to have_current_path(/\/instructions\?in_categories%5B%5D=#{category1.id}$/, url: true)
    
      # We make sure that only material1 is shown and material2 is not shown
      # The materials are filtered by category
      expect(page).to have_link material1.title
      expect(page).not_to have_link material2.title
    endT

    The spec is a Rails system spec. Other articles enter into more details about them. The point is:

    With a Rails system spec we don’t concern ourselves with JavaScript.

    We visit the page, we click, we see that something has changed on the page – like new materials were shown and others were hidden.

    What’s the use case for JavaScript specs then?

    Take a look at the following embedded BuildIn3D instruction. It is coming live and it has a next button. Click on the next button.

    TRS the Turning Radio Satellite construction from GeoSmart and in 3D

    Here is the actual spec in JavaScript with Jasmine and ran everyday with Teaspoon.

    it("right arrow button triggers TriedMoveIterator with 1", function(done) {
        // This is a JavaScript spec that is inside the browser. It has access to all the APIs 
        // of the browser. 
        const eventRight = new KeyboardEvent("keydown", { key: "ArrowRight" });
        document.onkeydown(eventRight);
        
        // Wait for a specific event to occur in the Instructions Steps (IS) framework
        // We are inside the browser here. There is no communication with the server
    
        IS.EventsUtil.WaitFor(() => this.iteratorListener.event).then(() => {
          expect(this.iteratorListener.event.getSteps()).toEqual(1);
          done();
        });
      });
    

    This the how we use JavaScript and this is why we need them:

    We need JavaScript specs that are run inside the browser to communicate with other JavaScript objects and the browser APIs for JavaScript apps.

    In this specific case there is this “iteratorListener” that monitors how the user follows the instructions. We need access to it. Otherwise it gets quite difficult to test. We can not write a spec to test what are the pixels on the screen that are drawn after clicking next. This will be … difficult to put it mildly. We don’t have to do it. We need to know that clicking the next button has triggered the proper action which will then draw the actual Geometry and Colors on the screen.

    How we use Jasmine and Teaspoon to run the JavaScript specs

    Years ago I found a tool called Teaspoon. Looking back this has been one of the most useful tools we’ve used in our stack. It allows us to have a Rails project (and we have a lot of those) and to run JavaScript specs in this Rails project. The best thing – it just works (the Rails way).

    # add teaspoon dependency
    $ cd test/dummy/
    $ rails generate teaspoon:install
    # write a few specs
    $ rails s -p 8889
    

    You start a server, visit the url in the browser and the specs are executed

    Tests are pure JavaScript and we use Jasmine.

    That’s it. Not much to add. It’s simple The specs are a simple JS file located in “spec/javascripts/” folder and here is an example for one of them

    describe("IS.EventDef", function() {
      describe("constructing throws error if", function() {
        it("event name is null", function() {
          expect(() => new IS.EventDef(null, {})).toThrow(new Error("Null argument passed."));
        });
    
        it("event conf is null", function() {
          expect(() => new IS.EventDef("smoe", null)).toThrow(new Error("Null argument passed."));
        });
    
        it("declaring an event, but the class attribute value is null", function() {
          expect(() => {
            const ext = IS.ExtensionDef.CreateByExtensionConf({
              extension: new IS.Extension(),
              events: {
                declaredEvent: {
                  class: null
                }
              }
            });
          }).toThrow("Declared event 'declaredEvent' class attribute is null!");
        });
      });
    });
    

     
  • kmitov 7:51 am on January 19, 2021 Permalink |
    Tags: , fllcasts, , , , ,   

    Same code – two platforms. With Rails. 

    (Everyday Code – instead of keeping our knowledge in an README.md let’s share it with the internet)

    We are running two platforms. FLLCasts and BuildIn3D. Both platforms are addressing entirely different problems to different sets of clients, but with the same code. FLLCasts is about eLearning, learning management and content management, while BuildIn3D is about eCommerce.

    What we are doing is running both platforms with the same code and this article is about how we do it. The main purpose is to give an overview for newcomers to our team, but I hope the community could benefit from it as a whole and I could get feedback and learn what others are doing.

    What do we mean by ‘same code’?

    FLLCasts has a route https://www.fllcasts.com/materials. Results are returned by the MaterialsController.

    BuildIn3D has a route https://platform.buildin3d.com/instrutions. Results are returned by the same MaterialsController.

    FLLCasts has things like Organizations, Groups, Courses, Episodes, Tasks which are for managing the eLearning part of the platform.

    BuildIn3D has none of these, but it has WebsiteEmbeds for the eCommerce stores to embed and put 3D building instructions and models on their E-commerce stores.

    We run the same code with small differences.

    Do we use branches?

    No, we don’t. Branches don’t work for this case. They are hard to maintain. We’ve tried to have an “fc_dev” branch and a “b3_dev” branch for the different platforms, but it gets difficult to maintain. You have to manually merge between the branches. It is true that Git has made merging quite easy, but still it is an “advanced” task and it is getting tedious when you have to do it a few times a day and to resolve conflicts almost every time.

    We use rails engines (gems)

    We are separating the platform in smaller rails engines.
    A common rails engine between FLLCasts and BuildIn3D is called fc-author_materials. It provides the functionality for and author to create a material both on FLLCasts and on BuildIn3D.

    The engine providing the functionality for Groups for the eLearning part of FLLCasts is called fc-groups. This engine is simply not installed on BuildIn3D, we install it only on FLLCasts.

    How does the Gemfile look like?

    Like this:

    install_if -> { !ENV.fetch('CAPABILITIES','').split(",").include?('--no-groups') } do
      gem 'fc-groups_enroll', path: 'gems/fc-groups_enroll'
      gem 'fc-groups', path: 'gems/fc-groups'
    end
    

    We call them “Capabilities”. By the default each platform is started with a “Capability” of having Groups. But we can disable them and tell the platform to start without Groups. When the platform starts the Groups are simply not there. We

    How about config/routes.rb?

    The fc-groups engine installs its own routes. This means that the main platform config/routes.rb is different from gems/fc-groups/config/routes.rb and the routes are installed only when the engine is installed.

    Another option is to have an if statement and to check for capabilities in the config/routes.rb. We still have to decide which is easier to maintain.

    Where do we keep the engines? Are they in a separate repo?

    We tried. We have a few of the engines in separate repos. With time we found out it is easier to keep them in the same repo.

    When the engines are in separate repos you have very strict dependencies between them. This proves to be useful but costs a lot in terms of development and creating a clear API between the engines. This could pay off when we would like to share the engines with the rest of the community like for example Refinery is doing. But, we are not there, yet. That’s why we found out we could spend the time more productively developing features instead of discussing which class goes where.

    With the case of all the rails engines in a single repo we have the mighty Monolith again, we have to be grown ups in the team and maintain it, but it is easier than having them in different repos.

    How do we configure the platforms?

    FLLCasts will send you emails from team [at] fllcasts [dot] com

    BuildIn3D will send you emails from team [at] buildin3d [dot] com

    Where is the configuration?

    The configuration is in the config/application.rb. The code looks exactly like this:

    platform = ENV.fetch('FC_PLATFORM', 'fc')
    if platform == 'fc'
      config.platform.sender = "team [at] fllcasts [dot] com"
    elsif platform == 'b3'
      config.platform.sender = "team [at] buildin3d [dot] com"
    

    When we run the platform we set and ENV variable called FC_PLATFORM. If the platform is “fc” this means “FLLCasts”. If the platform is “b3” this means “BuildIn3D”.

    In the config/environments/production.rb we are referring to Rails.application.config.platform.sender. In this way we have one production env for both platforms. We don’t have many production evns.

    Why not many production envs?

    We found out that if we have many production envs, we would also need many dev envs and many test envs and there will be a lot of duplication between them.

    That’s why we are putting the configuration in the application.rb. It’s about the application, not the environment.

    How do we deploy on heroku?

    First rule is – when you deploy one platform you also deploy the other platform. We do not allow different versions to be deployed. Both platforms are running the same code, always. Otherwise it gets difficult.

    When we deploy we do

    # In the same build we 
    # push the fllcasts app and then the buildin3d app
    git push fllcasts production3:master
    heroku run rake db:migrate --app fllcasts
    
    git push buildin3d production3:master
    heroku run rake db:migrate --app buildin3d

    In this way both platforms always share the same code, except for a short period of a few minutes between the deployment.

    How are the views separated?

    The platforms share a controller, but the views are different.

    The controller should return different views for the different platforms. Here is what the controller is doing.

    def show
       # If the platform is 'b3' return a different set of views
        if Rails.application.config.platform.id == "b3"
          render :template => Rails.application.config.platform.id+"/materials/show"
        else
          render :template => "/materials/show"
        end
      end

    In the same time we have the folders:

    # There are two separate folders for the views. One for B3 and one for FC 
    fc-author_materials/app/views/b3/materials/
    fc-author_materials/app/views/materials/

    How do we test?

    Testing proved to be challenging at first. Most of the time as the code is the same the specs should be the same right?

    Well, no. The code is the same, but the views are different. This means that the system specs are different. We write system and model specs and we don’t write views and controllers specs (if you are still writing views and controllers specs you should consider stopping. They were deprecated years ago).

    As the views are different the system specs are different.

    We tag the specs that are specifically for the BuildIn3D platform with a tag platform:b3

    
      context "platform is b3", platform: :b3 do
        before :each do
          expect_b3
        end

    When we run the specs we run first all the specs that are not specifically for b3 with

    $ rake spec SPEC="$specs_to_build" SPEC_OPTS="--tag ~platform:b3 --order random"

    Then we run a second suite for the tests that are specifically for the BuildIn3D platform.

    # Note that here we set an ENV and we use platform:b3 and not ~platform:b3
    $ FC_PLATFORM="b3" rake spec SPEC="$specs_to_build" SPEC_OPTS="--tag platform:b3 --order random"

    I see how this will become difficult to maintain if we start a third platform or a fourth platform, but we would figure it out when we get there. It is not something worth investing any resources into as we do not plan to start a new platform soon.

    Conclusion

    That’s how we run two platforms with the same code. Is it working? We have about 200 deployments so far in this way. We see that it is working.

    Is it difficult to understand? It is much easier than different branches and it is also much easier than having different repos.

    To summarize – we have a monolith app separated in small engines that are all in the same repo. When running a specific platform we install only the engines that we need. Controllers are the same, views could be different.

    I hope this was helpful and you can see a way to start a spin off of your current idea and to create a new business with the same code.

    There is a lot to be improved – it would be better to have each and every rails engine as a completely separate project in a different repo that we just include in the platform. But we still don’t have the requirement for this and it will require months of work on a 10 years old platform as ours. Once we see a clear path for it to pay off, we would probably do it in this way.

    For fun

    Thanks for stopping by and reading through this article. Have fun with this 3D model and building instructions.

    GeoShpere3 construction with GeoSmart set

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel