Tagged: rails Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 6:18 am on November 30, 2020 Permalink |
    Tags: rails, , ,   

    Testing an index page in a web application where we depend on the order. 

    [Everyday Code]

    Today the specs for an index page failed and I had to improve it. I decided to share a little trick for when we depend on the order of the objects.

    The specs is for the /subscriptions page where we show an index of all the subscriptions. We order the subscriptions by created_at in DESC. There are 20 subscriptions on the page. Then you must got to the next page. In the spec we open the page and check that there is a link to our subscription.

    visit "/admin/user/subscriptions"
    
    expect(page).to have_link subscription.random_id.to_s, href: "/admin/user/subscriptions/#{subscription.to_param}/edit"

    The problem was that there are other specs which for some reason create subscriptions into the future. This means that at a certain point in time when more than 20 subscriptions are created into the future in the DB, then our spec will fail. It will fail because the newly created subscription is on the second page.

    All the subscriptions on this page are into the future as today is 2020-11-30. So our newly created subscription is not here.

    What are our options?

    Move to the correct page in the spec

    This is an option. It will require us to have a loop in the spec that would more to the next page in the index and search for each subscription. This is too much logic for a spec.

    Delete all the future subscriptions before starting the spec

    Could be done. But is more logic for the spec. It actually needs to delete subscriptions and this is not the job of this specs.

    Create a subscription that is with the most recent created_at

    It is simple.

      let(:subscription) {FactoryBot.create(:subscription, created_at: subscription_created_at_time)}
    
      def subscription_created_at_time
        (Subscription.order(:created_at).last.try(:created_at) || Time.now)+1.second
      end
    
    

    It is the simpler change for this spec. Just create a subscription that’s last. In this way we know it will appear on the first page even when other specs have created subscriptions into the future.

     
  • kmitov 9:39 pm on November 28, 2020 Permalink |
    Tags: rails, , , ,   

    Why we should never clear our DB before/after running specs. 

    One common “mistake” I’ve seen a couple of times is to clean the Database before/after specs are run. It seems to be a common practice with reasonable arguments. I think this is a bad idea. Here is why and what we should do instead.

    Why is the DB cleared before/after the specs

    When running specs that need access to a DB we might have to create a User or an Article or a Project model, then connect them in a certain way and test the business logic of our spec. After the spec is finished it is not wise to delete these objects from the DB directly in the spec. Sometimes it takes additional time, sometimes it executes additional logic. In most cases you don’t clear the DB after each and every spec.

    It is a good idea to clean the db before all the specs or after all the specs if they are successful. In this way we reset the DB only once, it saves some time and is much cleaner because you can plug in this behavior if you want to.

    Why the DB should not be cleared before/after the specs

    The simple answer is that our code will never, absolutely never work on a clean db in a production. If we have a test procedure that runs the specs against a clean and empty db they might pass when the db is clean. But what use do we have from code that could work in a clean environment, but could not work in a real production environment. The answer is – non.

    We don’t clean our db before/after each spec. In this way we’ve been able to track some really nasty bugs. Like slow queries that are slow only when you have too many users. Other cases involve special relations that are built in time. Like users that are part of an organization and the organization was once having one check for uniqueness of the user and now it has another check. Because the db is not cleared every time we make sure that it is properly migrated with all the needed migrations.

    We found out that a 7 years out test db that is not cleared is closer to a 7 years old production db.

    The test db is not the production db

    The test db is not the production db. It might have the same scheme, that is for sure, but the amount of data in them and the complexity of this data is different. What we need is code that could run on a production db. There is no use of any code that could run only in test environment.

    So here is what we do:

    We export the production db, we change some data like user emails, names and any other sensitive data and we import it as a test db. We run the specs on this db.

    In this way we make sure that the code could actually run on a real db before deploying it.

     
  • kmitov 5:41 am on October 6, 2020 Permalink |
    Tags: , progressive web application, pwa, rails, ,   

    The path to a progressive web app – or how we skipped the whole JS Frameworks on the client thing. 

    I recently responded to a question in the Stimulus JS forum that prompted me to write this blog.

    About 6 months ago we decided to skip the whole “JSON response from server and JS framework on the client” stuff and we’ve never felt better. We significantly reduce the code base while delivering more features. We manage to do it with much less resources. Here is an outline of our path

    Progressive Web Application

    We had a few problems.

    1. Specs that were way to fragile and user experience that was way to fragile. Fragile specs that are failing from time to time are not a problem on their own. They are an indicator that the features also do not work in the client browsers from time to time.
    2. Too much and two difficult JS on the client. Although it might see that making a JSON request from the client to the server and generating the HTML from the response might seem as a good idea, it is not always a good idea. If we are to generate the HTML from a JSON, why don’t we ask the server for the HTML and be done with it? Yes, some would say it is faster, but we found out it is basically the same for the server to render ‘{“video_src”: “https://…&#8221;}’ or to render “<video src=’https://…&#8217;></video>’ . The drawback is that in the first scenario you must generate the video tag on the client and this means more work. Like twice the amount of work.

    So we said:

    Let’s deliver the platform to a browser that has NO JS at all, and if it has, we would enhance it here and there.

    How it worked out?

    In retrospective… best decision even. Just know that there is not JS in the browser and try to deliver your features. Specs got a lot faster and better. 1h 40 m compared to 31 minutes. They are not fragile. We have very little JS. The whole platform is much faster. We user one framework less. So, I see no drawbacks.

    First we made the decision not to have a JS framework on the client and to drop this idea as a whole. For our case it was just adding complexity and one more framework. This does not happen overnight, but it could happen. So we decide that there is no JS and the whole platform should work in the case of JS disabled on the browser (this bootstrap navigation menus are a pain in the a…). It should be a progressive web application (PWA).

    After this decisions we did not replace JSON with Ajax calls. We skipped most of them entirely. Some JSON requests could not be skipped, but we changed them as AJAX – for example “generating a username”. When users register they could choose a username, but to make it easier for them we generate one by default. When generating we must make sure it is a username that does not exists in the DB. For this we need to make a request to the server and this is one place we are using Stimulus to submit the username.

    A place that we still use JSON is with Datatables- it is just so convenient. There are also a few progress bars that are making some legacy JSON requests.

    Overall we have Ajax here and there, an a few JSON requests, but that’s it. Like 90-95% of the workflow is working with JS disabled.

    We even took this to the extreme. We are testing it with browsers with JS and browsers without JS. So a delete button on a browser without JS is not opening a confirmation. But with JS enabled the delete opens a confirmation. I was afraid this will introduce a lot of logic in the specs, but I am still surprised it did not. We have one method “js_agnostic_delete” with an if statement that check if JS is enabled and decides what to do.

    My point is that moving JSON to Ajax 1:1 was not for us. It would not pay off as we would basically be doing the same, but in another format. What really payed off and allowed us to reduce the code base with like 30-40%, increase the speed and make the specs not so fragile was to say – “let’s deliver our platform to a JS disabled browser, and if it has JS, than great.”

    To give you even more context this was a set of decisions we made in April 2020 after years of getting tired with JS on client. We are also quite experience with JS as we’ve build a pretty large framework for 3D that is running entirely in browser so it was not like a lack of knowledge and experience with JS on our side that brought us to these decisions. I think whole team grew up enough to finally do without JS.

     
  • kmitov 5:23 pm on May 21, 2020 Permalink |
    Tags: , , , rails, , , ,   

    With assets in a rails engine it could be Gemfile vs gemspec dependency that is messing it up 

    You have a rails engine. It has an asset (like a scss file). You include this engine in another engine. How do you test that the assets is correctly resolved?

    There is a moment when you simply can not understand where and how the asset is resolved/not resolved. The solution actually is in what are the Gemfile and .gemspec for a rails engine. Both of them could describe dependencies, but they contain different things.

    The whole premise of the situation seems strange until you get to building a rails engine that would contain the “theme” of your app. We go in this situation with two of our platforms – FLLCasts.com and 3DAssemblyInstructions.com. We wanted to have different layouts in different gems that are providing different look an feel- separate the main layout from dashboard and from main. We also wanted to have this layouts in a different gems so that we could release them with different versions and test them as separate gems in separate builds.

    Let’s get into this step by step tutorial and by the end I am sure you would know much about how assets are resolved, packed and testes when they are within a rails. Again I am writing this tutorial to help spread the knowledge internally, but the problem is so common that it might be of interest to the community as a whole.

    Case 1 – simple app with an asset

    Just to get the understanding correctly and to have a base we would create a simple rails app with an asset and start it. I am using rails 6 and ruby 2.6.5

    $ rails new simple_assets_app
    $ cd simple_asset_app
    # Create a new asset.scss file with some content
    $ echo "asset_style { background-color: white}" > app/assets/stylesheets/asset.scss
    $ rails s
    => Booting Puma
    => Rails 6.0.3.1 application starting in development 
    => Run `rails server --help` for more startup options
    Puma starting in single mode...
    * Version 4.3.5 (ruby 2.6.5-p114), codename: Mysterious Traveller
    * Min threads: 5, max threads: 5
    * Environment: development
    * Listening on tcp://127.0.0.1:3000
    * Listening on tcp://[::1]:3000
    Use Ctrl-C to stop
    
    

    In another terminal request the asset

    # Request the asset and dislay its content. Everything works. Perfect
    $ curl localhost:3000/assets/asset.scss
    asset_style { background-color: white}

    Case 2 – engine with an asset

    Now let’s build a rails plugin that would hold this asset

    $ rails plugin new simple_asset_engine --full
    $ cd simple_asset_engine/
    $ echo "asset_style { background-color: white}" > app/assets/stylesheets/asset.scss
    # You have to fix the TODOs in the gemspec. After you are ready do 
    $ rails s -p 3001
    => Booting WEBrick
    => Rails 6.0.3.1 application starting in development http://localhost:3001
    => Run `rails server --help` for more startup options
    [2020-05-21 19:48:51] INFO  WEBrick 1.4.2
    [2020-05-21 19:48:51] INFO  ruby 2.6.5 (2019-10-01) [x86_64-linux]
    [2020-05-21 19:48:51] INFO  WEBrick::HTTPServer#start: pid=13394 port=3001
    

    Now that the port is different.

    Request the asset

    # Nothing strange here. Asset is delivered as expected.
    $ curl localhost:3001/assets/asset.scss
    asset_style { background-color: white}
    

    Conclusion – engine, no engine the asset is delivered.

    Case 3 – require simple_asset_engine in another engine

    This could be a requirement. It has happen to us. It is a valid case. It is rather strange to have one engine require another, but we are all grown ups here…

    $ rails plugin new host_engine --full
    # Fix todos in .gemspec
    
    # Add a dependency to simple_asset_engine. Asset should be found, rigth
    $ echo 'gem "simple_asset_engine", path: "~/axles/tmp/simple_asset_engine"' >> Gemfile
    
    # install gems - mainly the simple_asset_engine
    $ bundle install
    
    # Start server. Note the different port
    $ rails s -p 4010
    
    

    Request asset

    $ curl localhost:4010/assets/asset.scss
    asset_style { background-color: white}

    Asset is found.

    That’s the power of rails. It just works. You can have an asset in an app, asset in an engine, asset in an engine, required in another engine and the asset is always found. Except…

    This might sound as a conclusion, because everything works. Until you get to production, which is actually the only time it matters if anything works, but that’s another story.

    Our current host_engine depends on a gem with a local path in the local filesystem.

    host_engine: $ cat Gemfile
    source 'https://rubygems.org'
    git_source(:github) { |repo| "https://github.com/#{repo}.git" }
    
    # Declare your gem's dependencies in host_engine.gemspec.
    # Bundler will treat runtime dependencies like base dependencies, and
    # development dependencies will be added by default to the :development group.
    gemspec
    
    # Declare any dependencies that are still in development here instead of in
    # your gemspec. These might include edge Rails or gems from your path or
    # Git. Remember to move these dependencies to your gemspec before releasing
    # your gem to rubygems.org.
    
    # To use a debugger
    # gem 'byebug', group: [:development, :test]
    gem "simple_asset_engine", path: "~/axles/tmp/simple_asset_engine"

    This is something that we can not ship to production. That’s why we pack “simple_asset_engine” as a gem and place this gem in a gemrepo. This is not part of the tutorial so I will simulate it.

    We should also modify the host_engine.gemspec to depend on simple_asset_engine. We add a dependency.

    # in host_engine.gemspec we add
    
    spec.add_dependency "simple_asset_engine"
    

    This means – ‘add a dependency to “simple_asset_engine”‘. Our gem will host_engine will depend on ‘simple_asset_engine’, but and here is a but, but only for ‘production’ environment. So this means – “production”. To simulate the fact that we don’t have this gem in a repo we would change the dependency in the Gemfile to be only for production.

    # This means the same as spec.add_dependency 'simple_asset_engine' It add the gem as a dependency to production. We are doing it like this to simulate that we have 'spec.add_dependency' and the gem is coming from a gems repo
    
    gem "simple_asset_engine", group: [:production], path: "~/axles/tmp/simple_asset_engine"

    Now as you do request the asset a strange error occurs:

    $ curl localhost:4010/assets/asset.scss
    <!DOCTYPE html>
    <html lang="en">
    <head>
      <meta charset="utf-8" />
      <title>Action Controller: Exception caught</title>
        ....
       <div id="Framework-Trace-0" style="display: none;">
          <code style="font-size: 11px;">
              <a class="trace-frames trace-frames-0" data-exception-object-id="47236752082100" data-frame-id="0" href="#">
                actionpack (6.0.3.1) lib/action_dispatch/middleware/debug_exceptions.rb:36:in `call'
              </a>
              <br>
              <a class="trace-frames trace-frames-0" data-exception-object-id="47236752082100" data-frame-id="1" href="#">
                actionpack (6.0.3.1) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
              </a>
              <br>
              <a class="trace-frames trace-frames-0" data-exception-object-id="47236752082100" data-frame-id="2" href="#">
                railties (6.0.3.1) lib/rails/rack/logger.rb:37:in `call_app'
              </a>
    ...
    </div>
    </body>
    </html>
    

    In the log you see

    ActionController::RoutingError (No route matches [GET] "/assets/asset.scss"):
      
    actionpack (6.0.3.1) lib/action_dispatch/middleware/debug_exceptions.rb:36:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
    railties (6.0.3.1) lib/rails/rack/logger.rb:37:in `call_app'
    railties (6.0.3.1) lib/rails/rack/logger.rb:26:in `block in call'
    activesupport (6.0.3.1) lib/active_support/tagged_logging.rb:80:in `block in tagged'
    activesupport (6.0.3.1) lib/active_support/tagged_logging.rb:28:in `tagged'
    activesupport (6.0.3.1) lib/active_support/tagged_logging.rb:80:in `tagged'
    railties (6.0.3.1) lib/rails/rack/logger.rb:26:in `call'
    sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:11:in `block in call'
    activesupport (6.0.3.1) lib/active_support/logger_silence.rb:36:in `silence'
    activesupport (6.0.3.1) lib/active_support/logger.rb:64:in `block (3 levels) in broadcast'
    activesupport (6.0.3.1) lib/active_support/logger_silence.rb:36:in `silence'
    activesupport (6.0.3.1) lib/active_support/logger.rb:62:in `block (2 levels) in broadcast'
    sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:11:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/request_id.rb:27:in `call'
    rack (2.2.2) lib/rack/method_override.rb:24:in `call'
    rack (2.2.2) lib/rack/runtime.rb:22:in `call'
    activesupport (6.0.3.1) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/static.rb:126:in `call'
    rack (2.2.2) lib/rack/sendfile.rb:110:in `call'
    actionpack (6.0.3.1) lib/action_dispatch/middleware/host_authorization.rb:82:in `call'
    railties (6.0.3.1) lib/rails/engine.rb:527:in `call'
    rack (2.2.2) lib/rack/handler/webrick.rb:95:in `service'
    

    It is basically telling us that an error occurred. An error also should occur. It is logical, it is the right thing to do, but it can take hours to debug and understand this.

    The error occurs because we are starting the rails server in ‘development’ environment and the dependency to ‘simple_asset_engine’ is only for production.

    Case 3.1 – change to production and development

    # Change host_engine/Gemfile to have
    
    gem "simple_asset_engine", group: [:production, :development], path: "~/axles/tmp/simple_asset_engine"
    

    Restart server on port 4010.

    host_engine: $ rails s -p 4010
    => Booting WEBrick
    => Rails 6.0.3.1 application starting in development http://localhost:4010
    => Run `rails server --help` for more startup options
    [2020-05-21 20:14:55] INFO  WEBrick 1.4.2
    [2020-05-21 20:14:55] INFO  ruby 2.6.5 (2019-10-01) [x86_64-linux]
    [2020-05-21 20:14:55] INFO  WEBrick::HTTPServer#start: pid=14167 port=4010

    Request the asset

    $ curl localhost:4010/assets/asset.scss
    asset_style { background-color: white}

    Now it is time for conclusion

    spec.add_dependency in a gemspec gives you a dependency for production. There is a ‘spec.add_development_dependency’ that exists, but there are great discussion about it here https://github.com/rubygems/rubygems/issues/1104 Read more there. Really. Read more.

    But as we are trying to test assets separately in an engine it is important to understand what Gemfile and gemspec could be used for. If an asset is not found from a dependency of a gemspec, probably the whole dependency is required for a different environment. In the same time if the dependency contains only assets, it might be difficult to get what is happening at first sight and is it your fault, or that the F..k is sprockets doing or event what the bigger f..k is webpacker doing. But for this case it is just plain old ruby dependencies that are messing us up.

     
  • kmitov 1:03 pm on April 22, 2020 Permalink |
    Tags: , , rails, , , ,   

    Rails 6 + webpacker + jquery + sprockets + jquery plugin (fancetree) 

    So you might be in the process of migrating to webpacker. This means your sprockets should continue working. This is difficult. Sprockets wants jquery available in the view and you don’t have jquery available in the views. You have it in the webpacker packs.

    These things won’t work

      <script> 
        $("element_id")
      </script>

    jQuery is only available to the PACKS it is not available to the VIEWS.

    But there is a solution – expose-loader

    Here is how to setup jquery to be available to the views in sprockets app that you are migrating to rails 6. I am starting from the previous article were we set up the project from 0. – https://kmitov.com/posts/rails-6-webpacker-yarn-fancytree-less/

    $ yarn add expose-loader

    Add configuration for exposing of jquery

    // config/webpack/environments.js
    const { environment } = require('@rails/webpacker')
     
    const less_loader= {
     test: /\.less$/,
     use: ['css-loader', 'less-loader']
    };
    environment.loaders.append('less', less_loader)
    
    +
    +const webpack = require('webpack')
    +// this makes jquery available in all pack and you don't
    +// have to import or require it each time 
    +environment.plugins.prepend(
    +  'Provide',
    +  new webpack.ProvidePlugin({
    +    $: 'jquery',
    +    jQuery: 'jquery'
    +  })
    +)
    +
    +// this exposes jquery to be available in the views
    +// <script>
    +//   console.log($('#tree'))
    +// </script>
    +environment.loaders.append('expose', {
    +  test: require.resolve('jquery'),
    +  use: [{
    +    loader: 'expose-loader',
    +    options: '$'
    +  }, {
    +    loader: 'expose-loader',
    +    options: 'jQuery',
    +  }]
    +})

    Also expose fancytree

    // config/webpack/environments.js
    ...
    +// this exposes fancytree to be available in the views
    +// <script>
    +//   console.log($('#tree').fancytree())
    +// </script>
    +environment.loaders.append('fancytree', {
    +  test: require.resolve('jquery.fancytree'),
    +  use: [{
    +    loader: 'expose-loader',
    +    options: 'fancytree'
    +  }]
    +})
    

    And you are done.

    How in your views you could do:

    <script>
      console.log($('#tree'))
      $(function(){
        $('#tree').fancytree({
          extensions: ['edit', 'filter'],
          source: [
            {title: "Node 1", key: "1"},
            {title: "Folder 2", key: "2", folder: true, children: [
              {title: "Node 2.1", key: "3"},
              {title: "Node 2.2", key: "4"}
            ]}
          ],
        });
        const tree = fancytree.getTree('#tree');
        // Note: Loading and initialization may be asynchronous, so the nodes may not be accessible yet.
      })
    </script>
    

    Fancytree as a jquery plugin is working in rails 6 views and is available also to sprockets compiled files.

     
  • kmitov 8:45 am on April 22, 2020 Permalink |
    Tags: css, , less, npm, rails, , , , yarn   

    Rails 6 + webpacker + yarn + Fancytree + LESS 

    We had to migrate a gem from using fancytree-rails as a ruby gem to a new rails 6 gem using webpacker and jquery.fancytree coming from npm. On top of that jquery.fancytree is using LESS (CSS) and you have to do a few configurations.

    App is available at https://github.com/thebravoman/rails6_webpacker_fancytree_less

    Here is how to do it it a few simple commands

    Table of contents

    Create a new rails project

    We want to have Fancrytree in this project.

       $ rails new project_with_less_and_fancytree
       $ cd project_with_less_and_fancytree
       $ rails g scaffold books
       $ rails db:migrate
     

    Add fancytree yarn package

    $ yarn add jquery.fancytree
       yarn add v1.22.4
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       info fsevents@1.2.12: The platform "linux" is incompatible with this module.
       info "fsevents@1.2.12" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       warning " > webpack-dev-server@3.10.3" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
       warning "webpack-dev-server > webpack-dev-middleware@3.7.2" has unmet peer dependency "webpack@^4.0.0".
       warning " > jquery.fancytree@2.35.0" has unmet peer dependency "jquery@>=1.9".
       [4/4] Building fresh packages...
       success Saved lockfile.
       success Saved 1 new dependency.
       info Direct dependencies
       └─ jquery.fancytree@2.35.0
       info All dependencies
       └─ jquery.fancytree@2.35.0
       Done in 3.29s.

    I like yarn.

    Add less and less-loader

    Later to include fancytree we would have to do things like

    import 'jquery.fancytree/dist/skin-lion/ui.fancytree.less'

    This means fancytree uses LESS. So we need to process this .less files. Oh, css, oh you evil you.

    yarn add less

    $ yarn add less
       yarn add v1.22.4
       [1/4] Resolving packages...
       warning less > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
       [2/4] Fetching packages...
       info fsevents@1.2.12: The platform "linux" is incompatible with this module.
       info "fsevents@1.2.12" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       warning " > jquery.fancytree@2.35.0" has unmet peer dependency "jquery@>=1.9".
       warning " > less-loader@5.0.0" has unmet peer dependency "webpack@^2.0.0 || ^3.0.0 || ^4.0.0".
       warning " > webpack-dev-server@3.10.3" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
       warning "webpack-dev-server > webpack-dev-middleware@3.7.2" has unmet peer dependency "webpack@^4.0.0".
       [4/4] Building fresh packages...
       success Saved lockfile.
       success Saved 4 new dependencies.
       info Direct dependencies
       └─ less@3.11.1
       info All dependencies
       ├─ asap@2.0.6
       ├─ image-size@0.5.5
       ├─ less@3.11.1
       └─ promise@7.3.1
       Done in 5.80s.

    yarn add less-loader

    You need less and less-loader

    $ yarn add less-loader
       yarn add v1.22.4
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       info fsevents@1.2.12: The platform "linux" is incompatible with this module.
       info "fsevents@1.2.12" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       warning " > jquery.fancytree@2.35.0" has unmet peer dependency "jquery@>=1.9".
       warning " > webpack-dev-server@3.10.3" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
       warning "webpack-dev-server > webpack-dev-middleware@3.7.2" has unmet peer dependency "webpack@^4.0.0".
       warning " > less-loader@5.0.0" has unmet peer dependency "less@^2.3.1 || ^3.0.0".
       warning " > less-loader@5.0.0" has unmet peer dependency "webpack@^2.0.0 || ^3.0.0 || ^4.0.0".
       [4/4] Building fresh packages...
       success Saved lockfile.
       success Saved 2 new dependencies.
       info Direct dependencies
       └─ less-loader@5.0.0
       info All dependencies
       ├─ clone@2.1.2
       └─ less-loader@5.0.0
       Done in 3.26s.
     

    Add less-loader to webpack environment

    They must be registered. Probably in another file, but here in enrovonments.js is fine this tutorial.

    // config/webpack/environments.js
    
    const { environment } = require('@rails/webpacker')
    
    // THIS IS THE NEW CODE
    const less_loader= {
      test: /\.less$/,
      use: ['css-loader', 'less-loader']
    };
    environment.loaders.append('less', less_loader)
    // END: THIS IS THE NEW CODE
    
    module.exports = environment

    Use fancytree

    Check out the documentation at https://github.com/mar10/fancytree/wiki#use-a-module-loader

    But basically you must require fancytree and use it.

    // NOTE: This seems to be working
    // app/javascripts/packs/application.js
    
    //... some other code.
    
    // THIS IS THE NEW CODE ADDED AT THE BOTTOM OF application.js
    // Import LESS or CSS:
    import 'jquery.fancytree/dist/skin-lion/ui.fancytree.less'
    
    const $ = require('jquery');
    
    const fancytree = require('jquery.fancytree');
    require('jquery.fancytree/dist/modules/jquery.fancytree.edit');
    require('jquery.fancytree/dist/modules/jquery.fancytree.filter');
    
    console.log(fancytree.version);
    
    $(function(){
      $('#tree').fancytree({
        extensions: ['edit', 'filter'],
        source: [
          {title: "Node 1", key: "1"},
          {title: "Folder 2", key: "2", folder: true, children: [
            {title: "Node 2.1", key: "3"},
            {title: "Node 2.2", key: "4"}
          ]}
        ],
      });
      const tree = fancytree.getTree('#tree');
      // Note: Loading and initialization may be asynchronous, so the nodes may not be accessible yet.
    })
    // END: THIS IS THE NEW CODE ADDED AT THE BOTTOM OF application.js

    NOTE – import is kind of not working

    There is another configuration at https://github.com/mar10/fancytree/wiki#use-a-module-loader but I could not make it work

    // NOTE: This is not working
    import 'jquery.fancytree/dist/skin-lion/ui.fancytree.less';  // CSS or LESS
    import {createTree} from 'jquery.fancytree';
    import 'jquery.fancytree/dist/modules/jquery.fancytree.edit';
    import 'jquery.fancytree/dist/modules/jquery.fancytree.filter';
    
    const tree = createTree('#tree', {
      extensions: ['edit', 'filter'],
      source: [
          {title: "Node 1", key: "1"},
          {title: "Folder 2", key: "2", folder: true, children: [
            {title: "Node 2.1", key: "3"},
            {title: "Node 2.2", key: "4"}
          ]}
        ],
    });
    // Note: Loading and initialization may be asynchronous, so the nodes may not be accessible yet.

    Start the application

    $ rails s
    => Booting Puma
    => Rails 6.0.2.2 application starting in development 
    => Run `rails server --help` for more startup options
    Puma starting in single mode...
    * Version 4.3.3 (ruby 2.6.5-p114), codename: Mysterious Traveller
    * Min threads: 5, max threads: 5
    * Environment: development
    * Listening on tcp://127.0.0.1:3000
    * Listening on tcp://[::1]:3000
    Use Ctrl-C to stop
    Started GET "/" for ::1 at 2020-04-22 09:20:06 +0300
       (0.4ms)  SELECT sqlite_version(*)
       (0.2ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
    Processing by Rails::WelcomeController#index as HTML
      Rendering /home/user/.rvm/gems/ruby-2.6.5/gems/railties-6.0.2.2/lib/rails/templates/rails/welcome/index.html.erb
      Rendered /home/user/.rvm/gems/ruby-2.6.5/gems/railties-6.0.2.2/lib/rails/templates/rails/welcome/index.html.erb (Duration: 17.4ms | Allocations: 471)
    Completed 200 OK in 45ms (Views: 25.4ms | ActiveRecord: 0.0ms | Allocations: 2931)
    
    
    Started GET "/books" for ::1 at 2020-04-22 09:20:09 +0300
    Processing by BooksController#index as HTML
      Rendering books/index.html.erb within layouts/application
      Book Load (0.2ms)  SELECT "books".* FROM "books"
      ↳ app/views/books/index.html.erb:13
      Rendered books/index.html.erb within layouts/application (Duration: 29.0ms | Allocations: 1230)
    [Webpacker] Compiling...
    [Webpacker] Compiled all packs in /home/user/axles/tmp/project_with_less_and_fancytree/public/packs
    [Webpacker] Hash: 32e57f147dbdcbbf0c82
    Version: webpack 4.43.0
    Time: 2269ms
    Built at: 04/22/2020 9:20:13 AM
                                         Asset       Size       Chunks                         Chunk Names
        js/application-bbe9c4a129ab949e0636.js    124 KiB  application  [emitted] [immutable]  application
    js/application-bbe9c4a129ab949e0636.js.map    139 KiB  application  [emitted] [dev]        application
                                 manifest.json  364 bytes               [emitted]              
    Entrypoint application = js/application-bbe9c4a129ab949e0636.js js/application-bbe9c4a129ab949e0636.js.map
    [./app/javascript/channels sync recursive _channel\.js$] ./app/javascript/channels sync _channel\.js$ 160 bytes {application} [built]
    [./app/javascript/channels/index.js] 211 bytes {application} [built]
    [./app/javascript/packs/application.js] 749 bytes {application} [built]
    [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 552 bytes {application} [built]
        + 3 hidden modules
    
    Completed 200 OK in 3998ms (Views: 3994.1ms | ActiveRecord: 0.7ms | Allocations: 23364)
    

    Open /books

    Visit http://localhost:3000/books. You should see no books

    Started GET "/books" for ::1 at 2020-04-22 09:23:56 +0300
    Processing by BooksController#index as HTML
      Rendering books/index.html.erb within layouts/application
      Book Load (0.2ms)  SELECT "books".* FROM "books"
      ↳ app/views/books/index.html.erb:13
      Rendered books/index.html.erb within layouts/application (Duration: 1.7ms | Allocations: 633)
    [Webpacker] Compiling...
    [Webpacker] Compilation failed:
    Hash: 60e4cd172f04061a66be
    Version: webpack 4.43.0
    Time: 4365ms
    Built at: 04/22/2020 9:24:02 AM
                                         Asset       Size       Chunks                         Chunk Names
        js/application-6ffd14b1620a1ad7ff96.js    717 KiB  application  [emitted] [immutable]  application
    js/application-6ffd14b1620a1ad7ff96.js.map    841 KiB  application  [emitted] [dev]        application
                                 manifest.json  364 bytes               [emitted]              
    Entrypoint application = js/application-6ffd14b1620a1ad7ff96.js js/application-6ffd14b1620a1ad7ff96.js.map
    [./app/javascript/channels sync recursive _channel\.js$] ./app/javascript/channels sync _channel\.js$ 160 bytes {application} [built]
    [./app/javascript/channels/index.js] 211 bytes {application} [built]
    [./app/javascript/packs/application.js] 1.52 KiB {application} [built]
    [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 552 bytes {application} [built]
        + 9 hidden modules
    

    Change books.html.erb

    Add a div element with id=tree

    Books

    <%# app/vies/books/index.html.erb %>
    <p id="notice"><%= notice %></p>
    
    <!-- THIS HERE IS WHAT WE ARE ADDING -->
    
    <div id="tree"></div>
    
    <!-- END: THIS HERE IS WHAT WE ARE ADDING -->
    
    <h1>Books</h1>
    
    <table>
      <thead>
        <tr>
          <th colspan="3"></th>
        </tr>
      </thead>
    
      <tbody>
        <% @books.each do |book| %>
          <tr>
            <td><%= link_to 'Show', book %></td>
            <td><%= link_to 'Edit', edit_book_path(book) %></td>
            <td><%= link_to 'Destroy', book, method: :delete, data: { confirm: 'Are you sure?' } %></td>
          </tr>
        <% end %>
      </tbody>
    </table>

    Final picture

    Showing how books index works with fancytree

    Errors that might occur

    No less-loader

    If no less loader is available the following could occur.

    Started GET "/books" for ::1 at 2020-04-22 08:52:40 +0300
       (0.1ms)  SELECT sqlite_version(*)
    Processing by BooksController#index as HTML
      Rendering books/index.html.erb within layouts/application
      Book Load (0.2ms)  SELECT "books".* FROM "books"
      ↳ app/views/books/index.html.erb:13
      Rendered books/index.html.erb within layouts/application (Duration: 2.1ms | Allocations: 762)
    [Webpacker] Compiling...
    [Webpacker] Compilation failed:
    Hash: 6210a48eff6aa0097a4c
    Version: webpack 4.43.0
    Time: 1464ms
    Built at: 04/22/2020 8:52:43 AM
                                         Asset       Size       Chunks                         Chunk Names
        js/application-8dcd2b9e8cc222d43650.js    718 KiB  application  [emitted] [immutable]  application
    js/application-8dcd2b9e8cc222d43650.js.map    841 KiB  application  [emitted] [dev]        application
                                 manifest.json  364 bytes               [emitted]              
    Entrypoint application = js/application-8dcd2b9e8cc222d43650.js js/application-8dcd2b9e8cc222d43650.js.map
    [./app/javascript/channels sync recursive _channel\.js$] ./app/javascript/channels sync _channel\.js$ 160 bytes {application} [built]
    [./app/javascript/channels/index.js] 211 bytes {application} [built]
    [./app/javascript/packs/application.js] 1.07 KiB {application} [built]
    [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 552 bytes {application} [built]
        + 9 hidden modules
    
    ERROR in ./node_modules/jquery.fancytree/dist/skin-lion/ui.fancytree.less 28:0
    Module parse failed: Unexpected token (28:0)
    File was processed with these loaders:
     * ./node_modules/less-loader/dist/cjs.js
    You may need an additional loader to handle the result of these loaders.
    |  * Helpers
    |  *----------------------------------------------------------------------------*/
    > .fancytree-helper-hidden {
    |   display: none;
    | }
     @ ./app/javascript/packs/application.js 19:0-59
    

    no less available

    If less was not installed this would happen

    Started GET "/books" for ::1 at 2020-04-22 09:26:54 +0300
    Processing by BooksController#index as HTML
      Rendering books/index.html.erb within layouts/application
      Book Load (0.1ms)  SELECT "books".* FROM "books"
      ↳ app/views/books/index.html.erb:13
      Rendered books/index.html.erb within layouts/application (Duration: 2.1ms | Allocations: 617)
    [Webpacker] Compiling...
    [Webpacker] Compilation failed:
    Hash: 1adef07918f113c9c28e
    Version: webpack 4.43.0
    Time: 1380ms
    Built at: 04/22/2020 9:26:56 AM
                                         Asset       Size       Chunks                         Chunk Names
        js/application-b032c274e5b1d8d383da.js    721 KiB  application  [emitted] [immutable]  application
    js/application-b032c274e5b1d8d383da.js.map    841 KiB  application  [emitted] [dev]        application
                                 manifest.json  364 bytes               [emitted]              
    Entrypoint application = js/application-b032c274e5b1d8d383da.js js/application-b032c274e5b1d8d383da.js.map
    [./app/javascript/channels sync recursive _channel\.js$] ./app/javascript/channels sync _channel\.js$ 160 bytes {application} [built]
    [./app/javascript/channels/index.js] 211 bytes {application} [built]
    [./app/javascript/packs/application.js] 1.52 KiB {application} [built]
    [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 552 bytes {application} [built]
        + 9 hidden modules
    
    ERROR in ./node_modules/jquery.fancytree/dist/skin-lion/ui.fancytree.less
    Module build failed (from ./node_modules/less-loader/dist/cjs.js):
    Error: Cannot find module 'less'
    Require stack:
    - /home/kireto/axles/tmp/pesho2/node_modules/less-loader/dist/index.js
    - /home/kireto/axles/tmp/pesho2/node_modules/less-loader/dist/cjs.js
    - /home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/loadLoader.js
    - /home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/NormalModule.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/NormalModuleFactory.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/Compiler.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/webpack.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack-cli/bin/utils/validate-options.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack-cli/bin/utils/convert-argv.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack-cli/bin/cli.js
    - /home/kireto/axles/tmp/pesho2/node_modules/webpack/bin/webpack.js
        at Function.Module._resolveFilename (internal/modules/cjs/loader.js:982:15)
        at Function.Module._load (internal/modules/cjs/loader.js:864:27)
        at Module.require (internal/modules/cjs/loader.js:1044:19)
        at require (/home/kireto/axles/tmp/pesho2/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
        at Object.<anonymous> (/home/kireto/axles/tmp/pesho2/node_modules/less-loader/dist/index.js:8:36)
        at Module._compile (/home/kireto/axles/tmp/pesho2/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
        at Module.load (internal/modules/cjs/loader.js:1002:32)
        at Function.Module._load (internal/modules/cjs/loader.js:901:14)
        at Module.require (internal/modules/cjs/loader.js:1044:19)
        at require (/home/kireto/axles/tmp/pesho2/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
        at Object.<anonymous> (/home/kireto/axles/tmp/pesho2/node_modules/less-loader/dist/cjs.js:3:18)
        at Module._compile (/home/kireto/axles/tmp/pesho2/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
        at Module.load (internal/modules/cjs/loader.js:1002:32)
        at Function.Module._load (internal/modules/cjs/loader.js:901:14)
        at Module.require (internal/modules/cjs/loader.js:1044:19)
        at require (/home/kireto/axles/tmp/pesho2/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
        at loadLoader (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/loadLoader.js:18:17)
        at iteratePitchingLoaders (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js:169:2)
        at iteratePitchingLoaders (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js:165:10)
        at /home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js:176:18
        at loadLoader (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/loadLoader.js:47:3)
        at iteratePitchingLoaders (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js:169:2)
        at runLoaders (/home/kireto/axles/tmp/pesho2/node_modules/loader-runner/lib/LoaderRunner.js:365:2)
        at NormalModule.doBuild (/home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/NormalModule.js:295:3)
        at NormalModule.build (/home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/NormalModule.js:446:15)
        at Compilation.buildModule (/home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/Compilation.js:739:10)
        at /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/Compilation.js:981:14
        at /home/kireto/axles/tmp/pesho2/node_modules/webpack/lib/NormalModuleFactory.js:409:6
     @ ./app/javascript/packs/application.js 20:0-59
    
    Completed 200 OK in 2801ms (Views: 2800.1ms | ActiveRecord: 0.1ms | Allocations: 5363)
    
     
  • kmitov 6:48 am on August 12, 2019 Permalink |
    Tags: rails, , ,   

    ‘if-else’ is Forbidden – why and how to reduce logic branching in your code. 

    Each time there is an if-else logic in your code there is some logic branching. A state is checked and based on the result from the check different path are taken. This means that two scenarios are implemented and two scenarios should be support from now on. This could quickly become a problem because the number of supported cases grows exponentially.

    The more decisions you make in the code the more cases you must support.

    At the end you start from Process1 but you have 4 different cases and path that you should support. These are 4 different tests that should be written and these are for different places where things could go wrong.

    The four different paths are Process 1,2,4; 1,2,5; 1,3,6; 1,3,7

    Demonstration of a live example and a real production bug

    We have Subscriptions in the platform. Each subscription has a max_users field that shows how many users you could add to this subscription. If max_users=1 this means you can have only one user using the subscription.

    When adding people to the subscription you have two cases

    1. You successfully add someone to the subscription and you show a message “Success”
    2. The max users is reached and you show a message “Error”

    The code in a simplified manner looks something like this:

    if subscription.save 
       form.show_message "Success"
    else 
       form.show_message "Error"

    While developing we’ve changed the code for the error from form.show_message “Error” to modal_dialog.show_message “Error”

    After that we’ve changed the implementation further and the code for modal_dialog.show_message “Error” was no longed called.

    As a result when the use tries to add someone to his subscription, for which they’ve payed, the app freezes and nothing happens. There is no error displayed and no user is added to the subscription.

    The bug occurred because with the latest changes we’ve forgot to manually check the second case of the if-else clause and there is was not test developed for this case.

    How to remove the if-else clause from this code

    subscription.save
    message = subscription.get_save_message
    form.show_message message

    The subscription.save knows what message to set based on whether the subscription was successfully saved. It could set Error or it could set Success. After that we just show whatever message the subscription has to the form. If it is empty that’s great. This means no errors have occurred. subscriptions.get_save_message could be implemented in many different ways. It could be on the subscription object or another object, but this depends on the technology and framework used. But after the method save is called the message is set and there is a single flow and now branches in our code. The method form.show_message is called a single time on a single place in our code. If we change the API of this method we would change in a single place and will not forget about the second place. There is always a single scenario. Save->Show message.




     
  • kmitov 7:06 am on January 22, 2019 Permalink |
    Tags: , rails,   

    Implementation of Multi-levels caching. Example for Rails 

    There are two difficult things in Computer Science. Choosing a name for a variable and cache invalidation.

    That being said I went on a journey to implement multi-levels caching in one of our platforms. Cache should be fast. Fast cache is expensive. If you can use 1M of very fast and expensive cache, why not implement a second level cache that is 10M, not that fast and not that expensive and 100M of normal cache that is cheap, but still faster than going to db.

    TL;DR;

    I decided to implement a DbCache that will store cached html rendered values directly in db and will access them from the DB instead of the fast Redis/Memcachier cache. All in the name of saving a few dollars on expensive fast cache that I do not really need.

    <% cache(cache_key) do # this is the redis cache %>
      <%= db_cache(cache_key) do # this is the db cache %>
        <% # actual calculation of value %>
      <% end %>
    <% end %>

    Implementation

    There is no need to constantly render the html/json of the content that we would like to serve to the client. It could be rendered once and served until updated. We are using Memcachier for a very fast access to cached values, but it is costing us more money. And in many cases we do not need this fast access.

    That’s why there is a DbCache implementation

    It works in the following way. It has a key and a value.

    When using in the view you can use

     <% cache(cache_key) do %>
       <%= db_cache(cache_key) do %>
       <% end %>
     <% end %>

    In this way if something is in cache we take it. This is L1 cache if you like. If it is not in L1 cache (that is stored in memory) than we ask db_cache. db_cache is our second level cache – L2. If the value is not in db_cache then we render it. The principle could be applied for L3 cache, although we are not there yet as a platform to need L3 cache.

    But it is db_cache. It is accessing the db. Why do you even call it cache?

    When the db_cache is accessed we make a single query to the db and retrieve a single record. For an indexed, not very large table this is fast. If the value is to be rendered again it will mean making a few more request for different objects and their associations and moving through all the trouble of rendering it again which involves views. By retrieving the HTML/JSON directly from DB we could directly serve it.

    How is db_cache implemented?

    DbCache model that stores the values in the db. It has a key and value columns. That’s it. Creating/retrieving/updating DbCache records is what is interesting.

    The key is column in the DB that is an integer. NOT a string. This integer is generated with a hash function and is than shifted right. The postgres db has a signed int column and the hash is generating an unsigned int. We have to shift because there is not enough space for storing unsigned int in a postgres db. In this way the cache key given from the developer is transformed to an internal key that is used for finding the record. And there of course is an index on this hash64 column.

    def db_cache key, &block
      # This will give us a 64 bit hash
      # Shift the value to reduce it because the column is signed and there is no room for
      # an unsigned value
      internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
      db_cache = DbCache.find_by(hash64: internal_key)
    
      ...
    end

    How are keys expired?

    If a key has expired we must create a new record for this key. Expiring keys could be difficult. Every record has an updated_at value. Every day a job on the db is run and if the updated_at value is more than specific days old it is automatically expired. This controls the number of records in the DB. I am kind of interested in storing only records that are regularly accessed. I think that if a page was not accessed in a couple of days, you generally do not need a cached value for it.

    This opens the next question:

    How are keys marked not to expire? If we change the accessed_at for a record on every read that will be slow because of a write to accessed_at

    True. It is important to expire old keys, but it is also important not to touch records on every read request because this will be very slow. If we make a touch on every request to the cache this will involve an update that will slow down the method. So an update is happening only once a day. See the db_cache.touch call below. The record could be accessed thousands of times today but there will be only one write to update the updated_at value. To touch the record.

    def db_cache key, &block
    
      internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
      db_cache = DbCache.find_by(hash64: internal_key)
      if db_cache.nil?
        # create cache value
      else
        db_cache.touch if db_cache.updated_at < Time.now - 1.day
      end
    
      ...
    end

    How fast is DbCache?

    These are just referenced values and of course this depends on the size of your cached values and the db and the load. In our specific case on Heroku we’ve found that the DbCache generally retrieves values in the range of 2 to 8 ms. In comparison the very fast Memcachier does this in the range of 2 to 8 ms.

    We also used NewRelic to look at the performance that end users are experiencing. And there was a large improvement because we could cache hundreds of MB of records in DB compared to the few MB for Memcachier that we are paying for.

    Rails specific details

    Since this code has to live in our platform and it is also bound to use some other rails object there are a few things more that I’ve done. Here is the full code that I hope gives a complete picture.

    # Author::    Kiril Mitov  
    # Copyright:: Copyright (c) 2018 Robopartans Group
    # License::   MIT
    module DbCacheHelper
    
      def db_cache key, &block
        # puts "db_cache: key: #{key}"
        result = nil
        if controller.respond_to?(:perform_caching) && controller.perform_caching
    
          # This will give us a 64 bit hash
          # Shift the value to reduce it because the column is signed and there is now room for
          # un unsigned value
          internal_key = Digest::XXH64.hexdigest(key).to_i(16) >> 1
    
          db_cache = DbCache.find_by(hash64: internal_key)
          if db_cache.nil?
            # puts "DBCache Miss: #{key}, #{internal_key}"
            Rails.logger.info "DBCache Miss: #{key}, #{internal_key}"
    
            content = capture(&block)
    
            # Use a rescue. This will make sure that if
            # a race condition occurs between the check for
            # existence of the db_cache and the actuall write
            # we will still be able to find the key.
            # This happens when two or more people access the site at exactly the
            # same time.
            begin
              # puts "DBCache: Trying to create"
              # puts "DBCache Count before find or create: #{DbCache.count}"
              db_cache = DbCache.find_or_create_by(hash64: internal_key)
              # puts "DBCache Count after find or create: #{DbCache.count}"
              # puts "DBCache: Found or record is with id:#{db_cache.id}"
            rescue ActiveRecord::RecordNotUnique
              retry
            end
    
            # The update is after the create because the value should not be part of the
            # create.
            db_cache.update(value: content)
          else
            # puts "DBCache Hit: #{key}, #{internal_key}"
            Rails.logger.info "DBCache Hit: #{key}, #{internal_key}"
            db_cache.touch if db_cache.updated_at < Time.now - 1.day
          end
    
          result = db_cache.value
        else
          result = capture(&block)
        end
        # Result could be nil if we've cached nil. So just dont return nil,
        # but return empty string
        result ? result.html_safe : ""
      end
    
    end


     
  • kmitov 2:02 pm on January 9, 2019 Permalink |
    Tags: , parallel, rails, ,   

    i18n locales and the pass of rspec parallel specs and 

    First rule of good unit testing is: each test should be independent of the other tests.

    But if there is a global variable like I18n.locale than one spec could touch it and another spec will be run in a different locale from the default.

    TL;DR;

    Before each suite of specs set the locale to the default. This ensures that the specs are run against the same locale each time. Specific code is:

    # spec/rails_helper.rb
    RSpec.configure do |config|
      config.before :each do
        I18n.locale = Globalize.locale = I18n.default_locale
      end
    end
    

    i18n breaks spec isolation

    Internationalization, or i18n, should be part of most platforms. This means that i18n features should be properly tests. In suites when one of the specs modifies i18n the specs that are run after that are dependent on this new local.

    This seem trivial, but we only go to explore it the moment we started running specs in parallel on different CPU cores. The specs were started and run in different times and order on each run.

     
  • kmitov 7:21 am on January 8, 2019 Permalink |
    Tags: , chromedriver, , google-chrome, rails, , ,   

    Chromedriver not filling all the whole password field in automated RSpec, Capybara, Feature Tests 

    This is such a lovely story. You will like it.

    When using google chromedriver to run capybara tests sometimes, just sometimes, especially if the tests are run in parallel, when the test has to fill a text field like a password, it fills only part of it. Last time checked for chromedriver 2.45

    TL; DR;

    Solution – google does not care, or it seems it is too difficult for the chromedriver team to resolve so there simply is no solution.

    What is the test?

    We are using Rails, Rspec, Capybara, Google chromedriver. We are developing feature tests. Tests are run in parallel with

    rake parallel:spec

    Here is the test in question. Simply fill a password on the form for upgrading a subscription, click on Confirm and expect to be redirected to a page that says – “You’ve upgraded your subscription”


    def submit_and_expect_success password
          # dialog opens to confirm with password
          fill_in "owner_password", with: password
          click_on "Confirm"
    
          expect_redirect_to "/subscriptions/#{subscription.to_param}"
    
          # If it redirects to this page, it means that the change was successful
          expect(page).to have_current_path "/subscriptions/#{subscription.to_param}"
    end

    And the tests are failing with timeout at expect_redirect_to. No, expect_redirect_to is a custom method, because we are using ActionCable to wait for subscription upgrade to finish. Because of the payment service at the back this sometimes takes a while and we want to show a nice progress and we need a websocket. But that being said the method is nothing special.

    module ExpectRedirect
      def expect_redirect_to url
        # If the path doesn't change before the timeout passes,
        # the test will fail, because there will be no redirect
    
        puts "expect url: #{url}"
        begin
          Timeout.timeout(Capybara.default_max_wait_time) do
            sleep(0.1) until url == URI(page.current_url).path
            page.current_url
          end
        rescue Timeout::Error=>e
          puts "Current url is still: #{page.current_url}"
          puts page.body
          raise e
        end
      end
    end

    If we are redirected to the url withing Capybara.default_max_wait_time than everything is fine. If not, we are raising the Timeout::Error.

    Parallel execution

    For some reason the test in question fails only when we are doing a parallel execution. Or at least mostly when we are doing parallel execution of the tests. So we moved through some nice articles to revise our understanding of Timeout again and again.

    https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/

    But nevertheless the tests were failing with Timeout::Error on waiting for a redirect and in the html we could see the error returned by the server:

    <div><p>Invalid password</p></div>

    How come the password is Invalid

    No this took a while to debug and it seems rather mysterious but this is what we got:

    User password in DB is: 10124ddeaf1a69e3748e308508d916b6

    The server receives from the html form: 10124ddeaf1a69e3748e30850

    User password in DB is: 74c2a3e926420e1a30363423f121fc1e

    The server receives from the html from: 74c2a3e926420e1a3

    and so on and so on.

    Sometimes the difference is 8 symbols. Sometimes it is 2. Sometimes it is 16.

    It seems to be a client side issue

    Like. JavaScript. If there is an error this strange it has to be in the JavaScript. Right. There in the javascript we see:

    let form_data = form.serializeObject();
    this.perform('start_change', form_data);

    The form gets serialized. Probably it is not serialized correctly. Probably the values that we are sending are just not the values on the form. So I revised my knowledge on serializing objects in JavaScript with

    https://stackoverflow.com/questions/17488660/difference-between-serialize-and-serializeobject-jquery

    So far so good. But the serialization was not the problem. Here is what I did. I fixed all the passwords to be 32 symbols.

    let form_data = form.serializeObject();
     if(form_data["owner_password"].lenght != 32) {
            form_data["owner_password"] = "this was:" + form_data["owner_password"] + " which is less than 32 symbols"
          }
    this.perform('start_change', form_data);

    It it happened. The value of the password field was simply not 32 symbols long. It was not filled during the test.

    A little bit of search and we arrive at:

    https://github.com/teamcapybara/capybara/issues/1890

    and there in the bottom of the issue, there is the standard: “Not our problem resolution” with the link to:

    https://bugs.chromium.org/p/chromedriver/issues/detail?id=1771&q=sendkeys&sort=-id&colspec=ID%20Status%20Pri%20Owner%20Summary

    It seems that google chromedriver is not filling all the characters in the password field. It is doing it on random and is completely unpredictable.

    Issue still exists on:

    Issue still exist for
    
    Chrome: Version 71.0.3578.98 (Official Build) (64-bit)
    Chromedriver chromedriver --version
    ChromeDriver 2.45.615279 (12b89733300bd268cff3b78fc76cb8f3a7cc44e5)
    Linux kireto-laptop3 4.4.0-141-generic #167-Ubuntu x86_64 GNU/Linux
    Description:	Ubuntu 16.04.5 LTS
    Release:	16.04
    Codename:	xenial

    Today we would try Firefox driver.


     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel