Tagged: rails Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 10:38 pm on January 13, 2022 Permalink |
    Tags: bootstrap, cssbundling-rails, jsbundling-rails, rails, rails 7,   

    Rails 7 with bootstrap 5 by cssbundling-rails with dart-sass and jsbundling-rails with esbuild. No webpack and webpacker and a salt of Stimulus 

    Today I put on my “new to Rails hat” and I decided to start a new Rails 7 project using the latest two new sharp knives that we have with Rails 7 to get a job done.

    The Job To Be Done is: Demonstrate Bootstrap 5 examples of a Toast in a Rails 7 project.

    This is a simple example of a popular Bootstrap 5 component that has both css and js involved. My goal is to build a website that will show a button and when the button is clicked a Toast will be visualised as it is in https://getbootstrap.com/docs/5.0/components/toasts/#live

    Provide sharp knives and the menu is omakase

    These are two points of Rails doctrine. The menu is omakase, which means that we have a good default for the tools that we are using. We are also given sharp knives to use – cssbundling-rails, jsbundling-rails, Stimulus, dart-sass, esbuild.

    The process

    I would like to bundle bootstrap 5 in the application. One of the options is to take bootstrap from npm and I would like to demonstrate how we bundle bootstrap from npm. Note that this might not be the right choice for you.

    NPM version

    Make sure you have npm 7.1+. I was trying for more than an hour to identify an error because my npm version was 6.X

    $ npm --version

    Rails 7 project

    # We pass in the option --css bootstrap that will configure most things for us
    $ rails _7.0.1_ new bProject --css bootstrap
    # enter the project folder
    $ cd bProject

    Everything is installed. You have all the dependencies to bootstrap, there is a node_modules folder containing node modules like bootstrap.

    Bundling the CSS

    cssbundling-rails gem is installed by default in the gem file.

    There is an app/assets/stylesheet/application.bootstrap.scss that imports the bootstrap css.

    It’s content is:

    @import 'bootstrap/scss/bootstrap';

    We would be using sass (again from npm) to build this .scss file. There is a script in the package.json

    {
      "name": "app",
      "private": "true",
      "dependencies": {
        "@hotwired/stimulus": "^3.0.1",
        "@hotwired/turbo-rails": "^7.1.0",
        "@popperjs/core": "^2.11.2",
        "bootstrap": "^5.1.3",
        "esbuild": "^0.14.11",
        "sass": "^1.48.0"
      },
      "scripts": {
        "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds",
        "build:css": "sass ./app/assets/stylesheets/application.bootstrap.scss ./app/assets/builds/application.css --no-source-map --load-path=node_modules"
      }
    }

    The script build:css will take the app/assets/stylesheets/application.bootstrap.scss and will produce an app/assets/builds/application.css

    The file application.css is the one that our application will be using. It is referred in application.html.erb as
    <%= stylesheet_link_tag “application”, “data-turbo-track”: “reload” %>

    <!DOCTYPE html>
    <html>
      <head>
        <title>BProject</title>
        <meta name="viewport" content="width=device-width,initial-scale=1">
        <%= csrf_meta_tags %>
        <%= csp_meta_tag %>
    
        <%= stylesheet_link_tag "application", "data-turbo-track": "reload" %>
        <%= javascript_include_tag "application", "data-turbo-track": "reload", defer: true %>
      </head>
    
      <body>
        <%= yield %>
      </body>
    </html>

    The live reload of CSS

    In order to change the css and live reload it we must start sass with –watch option as

    $ sass ./app/assets/stylesheets/application.bootstrap.scss ./app/assets/builds/application.css --no-source-map --load-path=node_modules --watch

    but don’t do this as there is a helper file that we should execute at the end of the article that is  – calling ./bin/dev

    Bundling the JavaScript

    jsbundling-rails is installed in the Gemfile.

    There is an app/javascript/application.js that imports bootstrap

    // Entry point for the build script in your package.json
    import "@hotwired/turbo-rails"
    import "./controllers"
    import * as bootstrap from "bootstrap"

    The application.js is bundled with esbuild as a tool  and the command is in the package json

    # part of package.json
    "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds",

    The result is produced in app/assets/builds

    The result is referred by the Rails Assets Pipeline and included in the html by calling javascript_inlcude_tag ‘application’ as in

    <!DOCTYPE html>
    <html>
      <head>
        <title>BProject</title>
        <meta name="viewport" content="width=device-width,initial-scale=1">
        <%= csrf_meta_tags %>
        <%= csp_meta_tag %>
    
        <%= stylesheet_link_tag "application", "data-turbo-track": "reload" %>
        <%= javascript_include_tag "application", "data-turbo-track": "reload", defer: true %>
      </head>
    
      <body>
        <%= yield %>
      </body>
    </html>

    Create a new controller

    Initial controller that will be the home screen containing the button.

    # app/controllers/home_controller.rb
    $ echo '
    class HomeController < ApplicationController
    
      def index
      end
    end
    ' > app/controllers/home_controller.rb

    Create a views folder

    # views folder
    $ mkdir app/views/home

    Create the view

    It contains a button and a toast.

    $ echo '
    <!-- app/views/home/index.html.erb -->
    <h1>Title</h1>
    <div data-controller="toast">
      <button type="button" class="btn btn-primary" data-action="click->toast#show">Show live toast</button>
    </div>
    
    <div class="position-fixed bottom-0 end-0 p-3" style="z-index: 11">
      <div id="liveToast" class="toast hide" role="alert" aria-live="assertive" aria-atomic="true">
        <div class="toast-header">
          <strong class="me-auto">Bootstrap</strong>
          <small>11 mins ago</small>
          <button type="button" class="btn-close" data-bs-dismiss="toast" aria-label="Close"></button>
        </div>
        <div class="toast-body">
          Hello, world! This is a toast message.
        </div>
      </div>
    </div>
    ' > app/views/home/index.html.erb

    Note the code:

    <div data-controller="toast">
      <button type="button" class="btn btn-primary" data-action="click->toast#show">Show live toast</button>
    </div>

    Here we have a ‘data-controller’ attribute in the div element and a ‘data-action’ attribute in the button element.
    This is how we will connect Stimulus JS as a javascript framework to handle the logic for clicking on this button.

    Configure the routes

    When we open “/” we want Rails to call the home#index method that will return the index.html.erb page.

    $ echo '
    # config/routes.rb
    Rails.application.routes.draw do
      # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html
    
      # Defines the root path route ("/")
     root "home#index"
    end' > config/routes.rb

    Create a new Stimulus controller

    We will use Stimulus as a small prefered framework in Rails for doing JavaScript. When the button in the above HTML fragment is selected the method show() in the controller will be called.

    # Use the helper method to generate a stimulus controller
    $ bin/rails generate stimulus toastController
    
    # create the content of the controller
    $ echo '
    // app/javascript/controllers/toast_controller.js
    import { Controller } from "@hotwired/stimulus"
    import { Toast } from "bootstrap"
    
    // Connects to data-controller="toast"
    export default class extends Controller {
    
      connect() {
      }
    
      show() {
        const toastElement = document.getElementById("liveToast")
        const toast = new Toast(toastElement)
        toast.show();
      }
    }
    ' > app/javascript/controllers/toast_controller.js

    Start the dev servers for rails, jsbundling-rails and cssbundling-rails

    $ ./bin/dev

    Visit localhost:3000

    There is a button that could be selected and a toast will appear.

     
  • kmitov 1:25 pm on December 19, 2021 Permalink |
    Tags: hugo, jamstack, rails   

    Classic challenge, new tool – addressing browser image caching issues with Hugo fingerprinting 

    Have you ever been on a support call with a service/vendor and they tell you – “Please, clear your browser cache!”

    Or you’ve been on the development side of a website and you upload a new image on your website only for users to continue reporting that they still see the old image. Then you have to ask all of your users to refresh their browser, which is not an easy thing to do.

    It’s a classic challenge with caching of assets like images and it generally has only one solution working in all cases.

    Recently we had the same challenge with the new website at BeMe.ai and I thought: “Let’s explore a tool from the perspective of this challenge”. The tool is a static website generator called Hugo and the job to be done is to address the image caching challenge so that our parents of autistic children always see the latest image on our website developed by our illustrator – Antonio.

    (image of a parent and a an autistic child. Used at https://beme.ai)

    In this article I will go through what the challenge is, why there is a caching challenge, how it could be addressed without any framework, with Rails (Ruby on Rails) and with Hugo. My hope is to give you a good understanding of how the Hugo team thinks, in what terms and in what direction. This is not a Hugo tutorial, but more how I’ve focused on one challenge and tried to address it with Hugo.

    The challenge of browser caching

    When browsers visualise a webpage with images they request images from the server where the website is hosted. Images require much more bandwidth then text. Next time you visit the same website the browser will request the same image that will again require traffic. Images do not change often on websites so browsers prefer to cache the image – which means they are saved on the local storage of your device, be it mobile or desktop. Next time when you as a user open the same webpage the browser will not make a request to the server to get the same image as there is a copy of this same image already in your local storage.

    The question here is how does the browser know that the image it has in its local storage is the same image as the one that is on the server. What if you’ve changed the image in the meantime and there is a new version on the server. It happened with us and an image that Antonio developed.

    The answer is remarkably simple: The browser knows if the image in its local storage and the image on the server are the same if they have the same “url/name”.

    Let me illustrate a scenario

    A week ago:
    A parent visited our website. The browser visualised the image called – parent-with-child.png
    It stores a copy of parent-with-child.png in its local storage.

    Image from BeMe.ai that had the wrong dashboard on the phone

    Two days ago:
    Antonio developed a new image of the parent and the child and we uploaded it at BeMe.ai. From now on the image located on parent-with-child.png is the second version.

    Image improved by Antonio to have the correct dashboard on the phone. Taken from the BeMe.ai website.

    Today:
    The same parent again visits the website. The browsers asked the server what’s on the page and the server responded that the page contains a link to parent-with-child.png. As the browser already has a local copy of the parent-with-child.png it will not request this resource from the server. It will just use the local copy. This saves the bandwidth and the site is opened faster. It’s a better experience, but it is the old image.

    Which one will be shown to the user:

    What really makes this problem difficult is the fact that different browsers will behave in different ways. The internet has tried many different solutions including different headers in the HTTP protocol to address this challenge. Still, there are times when the user will just not see the new version of the image. It could drive you really crazy as it will be some users seeing the new version and some seeing the old.

    How big of a challenge is this?

    Technically it is not a great challenge, yet I’ve seen experienced engineers miss this small detail. I’ve missed it a couple of times. Every such case is a load on the support channel of your organisation and on the engineering. So better avoid any caching issues at all.

    What’s the solution?

    There are many solutions. There is only one that works across all browsers and devices and across all versions of the HTTP protocol that power the internet.
    The solution is remarkably simple:
    The browser will look at the URL of the image. If there is a local copy stored for this URL, the browser will use the local copy. If there is no local copy for the URL the browser will request the new image.

    Everything we have to do is next time we change the parent-with-child.png image to upload the image with a new file name. Probably parent-with-child-v2.png is a good new name.
    Other good names include:

    1. parent-with-child-2021-12-19-13-15.png (it has the date and time on which it was uploaded)
    2. parent-with-child-with-an-added-book.png (it is different and descriptive)
    3. parent-with-child-1639913778.png (it has in its name the time in seconds since the UNIX epoch when the file was created)
    4. parent-with-child-2f11cf241d023448c988c3fc807e7678.png (it has an MD5 hash code)
    5. parent-with-child-a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56.png (it has a SHA256 sum as it’s name)

    All it takes to resolve the browser caching challenge is to change the name of the picture when you upload it and to change it to something unique.

    That’s all it takes – no need for frameworks and fingerprinting and assets precompilation and all the other things that follow in the article.

    All it takes to address and successfully resolve any browser image caching issue is to change the name of the image next time you upload a new version of it.

    Why does the simple solution not work?

    I think it does not work because we are humans and we forget. We create a new version of the parent-with-child.png image and we forget to change the name of the image. That’s it. We just forget.

    Computers on the other hand are good at reminding us of things and of doing dull work that we often forget to do. What we could ask the computer to do is to create a new name for the image everytime we upload a new version. Enter fingerprinting?

    Fingerprinting

    Fingerprinting is the process of looking at the bits of the image (or generally any file) and calculating a checksum. The checksum will be unique for every file. After we calculate the checksum we add the checksum to the name of the file.

    Example:

    1. We upload the original parent-with-child.png image and the computer calculates a checksum a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56. Then it sets the name of the file to be parent-with-child-a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56.png on the server
    2. We upload a new version of parent-with-child.png image and the computer calculates its checksum that is cdc72fa95ca711271e9109b8cab292239afe5d1ba3141ec38b26d9fc8768017b. Then the computers sets the name of the file to be parent-with-child-cdc72fa95ca711271e9109b8cab292239afe5d1ba3141ec38b26d9fc8768017b.png on the server
    3. We upload a new version, a new checksum is calculated and new name is generated. And this is done with every new image.

    How checksums are calculated is a topic for another article. Computers are good and fast at calculating checksum. Humans are terrible. Like literally it will probably take us days of manual calculations to come up with the checksum of an image file if we do it by hand.

    What’s difficult with fingerprinting?

    The difficult part is not the fingerprinting part. What’s difficult is finding every HTML page on your website where the image
     parent-with-child-a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56.png is used and replacing this name with parent-with-child-cdc72fa95ca711271e9109b8cab292239afe5d1ba3141ec38b26d9fc8768017b.png.

    This means every occurance of on the website

    <img src="https://www.beme.ai/images/parent-with-child-a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56.png"></img>

    should be updated to contain the new url

    <img src="https://www.beme.ai/images/parent-with-child-cdc72fa95ca711271e9109b8cab292239afe5d1ba3141ec38b26d9fc8768017b.png"></img>

    The good thing is that computers are also good at this – searching through the content of many files and replacing specific parts of this file with new content.

    What needs to be implemented then? What’s the process?

    What we need from our process of deploying new versions of images to our website is the following:

    1. Ask our illustrator, Antonio in our case, to develop a new version of the parent-and-child.png image.
    2. Put this new picture on our website and the computer should magically:
        – calculate a new checksums of the image and change the name of the parent-and-child.png, eg. fingerprint the image
        – find all references on our website to the previous version of the image and replace each reference with the new name of the image
       

    Pure Bash and Linux implementations

    Linux provides simple tools as sha256sum, grep, sed, mv and in a combination with such tools we can come up with a pretty decent solution. I am not going to do that because we might decide it is a good idea to do it in this way. This might take us on a path where we are reinventing the wheel with different bash scripts all over the infrastructure and code, and there is no need to do this. If you are already on this path I can not stop you, but I don’t want to be the one guiding you on this path. Been there, done that and after many years I realised that it was not a very wise decision.

    Doing it the Rails way

    I am a big fan of Rails. Rails address the browser image caching challenge with something called “Assets Pipeline”

    When we do in rails is to use the image_tag method in all HTML pages. The syntax below is an ERB template where we use “<%  %>” inside the HTML when processing it on the server side.

    <div class="container">
      <!-- Logo -->
      <%= link_to image_tag("logo.png", alt: "BeMe.ai", class: "img-fluid", width:55), main_app.root_path,{ class: "navbar-brand" }%>
      <!-- End Logo -->

    Note that here we use the name of “logo.png” and the image_tag method handles everything for us

    <div class="container">
      <a class="navbar-brand" href="/"><img alt="BeMe.ai" class="img-fluid" width="55" src="/assets/logo-60ffa36d48dfd362e6955b36c56058487272e3030d30f0b6b40d226b8e956a2b.png"></a>

    Note how the file that we referred as logo.png in the template now becomes /assets/logo-60ffa36d48dfd362e6955b36c56058487272e3030d30f0b6b40d226b8e956a2b.png in the file visualised on the client.

    Rails has done everything to us – fingerprinting and replacing. Thanks to the Assets Pipeline in Rails we’ve successfully resolved the browser image caching challenge.

    Doing it the Hugo way

    Hugo is different from Rails. Hugo is a static website generator and it thinks in terms different from Rails. Yet, it has a Hugo Pipeline. Before we enter into the Hugo Pipeline it is good to have a small introduction to Hugo.

    Hugo allows authors to create markdown documents

    Hugo is thinking about the authors. It gives a chance to include team members to develop the content of the website and it does not have to be team members that know how to start & deploy Rails applications. Which is good.

    This means that authors could create a markdown document like this

    # Basic misconceptions about autism
    
    In this blog post we will talk about basic misconceptions about autism.
    
    Let's start with this picture of an autistic child and a parent
    
    ![Picture of misconceptions](https://www.beme.ai/parent-and-child.png)
    
    ...

    As a markdown document this is something that someone with medical knowledge could develop the content and there is no need to have someone with medical and HTML/Rails/Web Development expertise (and such people are difficult to find)

    Now the author has added a new version of the parent-and-child.png image and it again has the name parent-and-child.png. We should somehow ask Hugo to add a fingerprint and replace all the references to the image with a reference to the new image.

    Hugo in 1 paragraph – content, layouts, markdown hooks

    In Hugo the content developers write the content in Markdown format. The engineer creates the HTML layouts. Hugo takes the layout and adds the content to the layout to generate the HTML page that should be visualised to the user. Everytime a Markdown element is converted to an HTML, Hugo calls a Markdown hook. The job of the hook is to convert the Markdown to an HTML. The logic of the hook is implemented with the Go Template Language. There are default implementations of hooks for every markdown element. We can override the default implementation of the hook that converts the Markdown containing the image parent-and-child.png to an HTML, by creating a file layouts/_default/_markup/render-image.html

    Fingerprinting of content images in Hugo with the Hugo Pipeline

    Fingerprinting of content images is not enabled by default. We should be explicit that we want it. Hugo Pipeline handles the rest with methods like “fingerprint”

    Here is the content of layouts/_default/_markup/render-image.html

    <!-- layouts/_default/_markup/render-image.html -->
    
    {{/* Gets a resource object from the destination of the image that is parent-with-child.png */}}
    {{/* The parent-with-child.png image is specified in the markdown and must be available in the  */}}
    {{/* asset folder */}}
    {{ $image := resources.GetMatch .Destination }}
    
    {{/* Calculate the fingerprint, add it to the name and get the Permalink is the full URL of the file */}}
    {{ $image := ($image | fingerprint).Permalink }}
    
    <img src="{{ $image }}"/>

    When processed Hugo will generate an index.html file that contains:

    <img src="http://example.org/parent-and-child.a6e0922d83d80151fb73210187a9fb55ee2c5d979177c339f22cb257dead8a56.png"

    Summary

    Image fingerprinting is guaranteed to resolve the browser caching challenge 100% of the cases.
    It is a topic that is often overlooked, both by content developers and engineers.
    Without it we often end up with users seeing the wrong images and having the “Clear their browser cache and refresh again.”
    It is easy to address it with many different available tools.

    We’ve looked at how to implemented it with
      – Linux
      – Rails
      – Hugo
     
    Asking users to “clear browser cache” and “refresh a website” is a failure of the process and the engineering organisation. It should not happen, and I am sure we could be better than this.

     
  • kmitov 8:57 am on October 8, 2021 Permalink |
    Tags: amazon-s3, rails,   

    Sometimes you need automated test on production 

    In this article I am making the case that sometimes you just need to run automated tests against the real production and the real systems with real data for real users.

    The case

    We have a feature on one of our platforms:

    1. User clicks on “Export” for a “record”
    2. A job is scheduled. It generates a CSV file with information about the record and uploads on S3. Then a presigned_url for 72 hours is generated and an email is sent to the user with a link to download the file.

    The question is how do you test this?

    Confidence

    When it comes to specs I like to develop automated specs that give me the confidence that I deliver quality software. I am not particularly religious to what the spec is as long as it gives me confidence and it is not standing in my way by being too fragile.

    Sometimes these specs are model/unit specs, many times they are system/feature/integration specs, but there are cases where you just need to run a test on production against the production db, production S3, production env, production user, production everything.

    Go in a System/Integration spec

    A spec that would give me confidence here is to simulate the user behavior with Rails system specs.
    The user goes and click on the Export and I check that we’ve received an email and this email contains a link

      scenario "create an export, uploads it on s3 and send an email" do
        # Set up the record
        user = FactoryBot.create(:user)
        record = FactoryBot.create(:record)
        ... 
    
        # Start the spec
        login_as user
        visit "/records"
        click_on "Export"
        expect(page).to have_text "Export successfully scheduled. You will receive an email with a link soon."
    
        mail_html_content = ActionMailer::Base.deliveries.select{|email| email.subject == "Successful export"}.last.html_part.to_s
        expect(mail_html_content).to have_xpath "//a[text()='#{export_name}']"
        link_to_exported_zip = Nokogiri::HTML(mail_html_content).xpath("//a[text()='#{export_name}']").attribute("href").value
    
        csv_content = read_csv_in_zip_given_my_link link_to_exported_zip 
        expect(csv_content).not_to be_nil
        expect(csv_content).to include user.username
      end

    This spec does not work!

    First problem – AWS was stubbed

    We have a lot of other specs that are using S3 API. It is a good practice as you don’t want all your specs to touch S3 for real. It is slow and it is too coupled. But for this spec there was a problem. There was a file uploaded on S3, but the file was empty. The reason was that on one of the machines that was running the spes there was no ‘zip’ command. It was not installed and we are using ‘zip’ to create a zip of the csv files.

    Because of this I wanted to upload an actual file somehow and actually check what is in the file.

    I created a spec filter that would start a specific spec with real S3.

    # spec/rails_helper.rb
    RSpec.configure do |config|
      config.before(:each) do
        # Stub S3 for all specs
        Aws.config[:s3] = {
          stub_responses: true
        }
      end
    
      config.before(:each, s3_stub_responses: false) do
        # but for some specs, those that have "s3_stub_responses: false" tag do not stub s3 and call the real s3.
        Aws.config[:s3] = {
          stub_responses: false
        }
      end
    end

    `This allows us to start the spec

      scenario "create an export, uploads it on s3 and send an email", s3_stub_responses: false do
        # No in this spec S3 is not stubbed and we upload the file
      end

    Yes, we could create a local s3 server, but then the second problem comes.

    Mailer was adding invalid params

    In the email we are sending a presigned_url to the S3 file as the file is not public.
    But the mailer that we were using was adding “utm_campaign=…” to the url params.
    This means that the S3 presigned url was not valid. Checking if there is an url in the email was simply not enough. We had to actually download the file from S3 to make sure the link is correct.

    This was still not enough.

    It is still not working on production

    All the tests were passing with real S3 and real mailer in test and development env, but when I went on production the feature was not working.

    The problem was with the configuration. In order to upload to S3 we should know the bucket. The bucket was configured for Test and Development but was missing for production

    config/environments/development.rb:  config.aws_bucket = 'the-bucket'
    config/environments/test.rb:  config.aws_bucket = 'the-bucket'
    config/environments/production.rb: # there was no config.aws_bucket

    The only way I could make sure that the configuration in production is correct and that the bucket is set up correctly is to run the spec on a real production.

    Should we run all specs on a real production?

    Of course not. But there should be a few specs for a few features that should test that the buckets have the right permissions and they are accessible and the configuration in production is right. This is what I’ve added. Once a day a spec goes on the production and tests that everything works on production with real S3, real db, real env and configuration, the same way that users will use the feature.

    How is this part of the CI/CD?

    It is not. We do not run this spec before deploy. We run all the other specs before deploy that gives us 99% confidence that everything works. But for the one percent we run a spec once every day (or after deploy) just to check a real, complex scenario, involving the communication between different systems.

    It pays off.

     
  • kmitov 7:33 am on October 8, 2021 Permalink |
    Tags: hotwire, , rails, turbo   

    [Rails, Hotwire] Migrate to Turbo from rails-ujs and turbolinks – how it went for us. 

    We recently decided to migrate one of our newest platforms to Turbo. The goal of this article is to help anyone who plans to do the same migration. I hope it gives you a perspective of the amount of work required. Generally it was easy and straightforward, but a few specs had to be changed because of urls and controller results

    Gemfile

    Remove turbolinks and add turbo-rails. The change was

    --- a/Gemfile.lock
    +++ b/Gemfile.lock
    @@ -227,9 +227,8 @@ GEM
         switch_user (1.5.4)
         thor (1.1.0)
         tilt (2.0.10)
    -    turbolinks (5.2.1)
    -      turbolinks-source (~> 5.2)
    -    turbolinks-source (5.2.0)
    +    turbo-rails (0.7.8)
    +      rails (>= 6.0.0)

    application.js and no more rails-ujs and Turbolinks

    Added “@notwired/turbo-rails” and removed Rails.start() and Turbolinks.start()

    --- a/app/javascript/packs/application.js
    +++ b/app/javascript/packs/application.js
    @@ -3,8 +3,7 @@
     // a relevant structure within app/javascript and only use these pack files to reference
     // that code so it'll be compiled.
    
    -import Rails from "@rails/ujs"
    -import Turbolinks from "turbolinks"
    +import "@hotwired/turbo-rails"
     import * as ActiveStorage from "@rails/activestorage"
     import "channels"
    
    @@ -14,8 +13,6 @@ import "channels"
     // Collapse - needed for navbar
     import { Collapse } from 'bootstrap';
    
    -Rails.start()
    -Turbolinks.start()
     ActiveStorage.start()

    package.json

    The change was small

    --- a/package.json
    +++ b/package.json
    @@ -2,10 +2,10 @@
       "name": "platform",
       "private": true,
       "dependencies": {
    +    "@hotwired/turbo-rails": "^7.0.0-rc.3",
         "@popperjs/core": "^2.9.2",
         "@rails/actioncable": "^6.0.0",
         "@rails/activestorage": "^6.0.0",
    -    "@rails/ujs": "^6.0.0",
         "@rails/webpacker": "5.4.0",
         "bootstrap": "^5.0.2",
         "stimulus": "^2.0.0",

    Device still does not work

    For the device forms you have to add “data: {turbo: ‘false’}” to disable turbo for them

    +<%= form_for(resource, as: resource_name, url: password_path(resource_name), html: { method: :post }, data: {turbo: "false"}) do |f| %>;

    We are waiting for resolutions on https://github.com/heartcombo/devise/pull/5340

    Controllers have to return an unprocessable_entity on form errors

    If there are active_record.errors in the controller we must now return status: :unprocessable_entity

    +++ b/app/controllers/records_controller.rb
    @@ -14,7 +14,7 @@ class RecordsController < ApplicationController
         if @record.save
           redirect_to edit_record_path(@record)
         else
    -      render :new
    +      render :new, status: :unprocessable_entity
         end
       end

    application.js was reduced significantly

    The old application.js – 923 KiB

      application (932 KiB)
          js/application-dce2ae8c3797246e3c4b.js

    The new application.js – 248 KiB

    remote:        Assets: 
    remote:          js/application-b52f4ecd1b3d48f2f393.js (248 KiB)

    Conclusion

    Overall a good experience. We are still facing some minor issues with third party chat widgets like tawk.to that do not work well with turbo, as they are sending 1 more request, refreshing the page and adding the widget to an iframe that is lost with turbo navigation. But we would probably move away from tawk.to.

     
  • kmitov 6:29 am on October 8, 2021 Permalink |
    Tags: , , rails,   

    [Rails] Warden.test_reset! does not always reset and the user is still logged in 

    We had this strange case of a spec that was randomly failing

      scenario "generate a subscribe link for not logged in users" js: true do 
        visit "/page_url"
    
        expect(page).to have_xpath "//a[text()='Subscribe']"
        click_link "Subscribe"
        ...
      end 

    When a user is logged in we generate a button that subscribes them immediately. But when a user is not logged in we generate a link that will direct the users to the subscription page for them to learn more about the subscription.

    This works well, but the spec is randomly failing sometimes.

    We expect there to be a link, eg. “//a” but on the page there is actually a button, eg. “//button”

    What this means is that when the spec started there was a logged in user. The user was still not logged out from the previous spec.
    This explains why sometimes the spec fails and why not – because we are running all the specs with a random order

    $ RAILS_ENV=test rake spec SPEC='...' SPEC_OPTS='--order random'

    Warden.test_reset! is not always working

    There is a Warden.test_reset! that is supposed to reset the session, but it seems for js: true cases where we have a Selenium driver the user is not always reset before the next test starts.

    # spec/rails_helper.rb
    RSpec.configure do |config|
      ...
      config.after(:each, type: :system) do
        Warden.test_reset!
      end
    end

    Logout before each system spec that is js: true

    I decided to try to explicitly log out before each js: true spec that is ‘system’ so I improved the RSpec.configuration

    RSpec.configure do |config|
      config.before(:each, type: :system, js: true) do
        logout # NOTE Sometimes when we have a js spec the user is still logged in from the previous one
        # Here I am logging it out explicitly. For js it seems Warden.test_reset! is not enough
        #
        # When I run specs without this logout are
        # Finished in 3 minutes 53.7 seconds (files took 28.79 seconds to load)
        #   383 examples, 0 failures, 2 pending
        #
        # With the logout are
        #
        # Finished in 3 minutes 34.2 seconds (files took 21.15 seconds to load)
        #   383 examples, 0 failures, 2 pending
        #
        # Randomized with seed 26106
        # 
        # So we should not be losing on performance
      end
    end

    Conclusion

    Warden.test_reset! does not always logout the user successfully before the next spec when specs are with Selenium driver – eg. “js: true”. I don’t know why, but that is the observed behavior.
    I’ve added a call to “logout” before each system spec that is js: true  to make sure the user is logged out.

     
  • kmitov 5:48 am on September 6, 2021 Permalink |
    Tags: , rails, , ,   

    Refresh while waiting with RSpec+Capybara in a Rails project 

    This is some serious advanced stuff here. You should share it.

    A colleague, looking at the git logs

    I recently had to create a spec with Capybary+RSpec where I refresh the page and wait for a value to appear on this page. It this particular scenario there is no need for WebSockets or and JS. We just need to refresh the page.

    But how to we test it?

    # Expect that the new records page will show the correct value of the record
    # We must do this in a loop as we are constantly refreshing the page.
    # We need to stay here and refresh the page
    # 
    # Use the Tmeout.timeout to stop the execution after the default Capybara.default_max_wait_time
    Timeout.timeout(Capybara.default_max_wait_time) do
      loop do
        # Visit the page. If you visit the same page a second time
        # it will refresh the page.
        visit "/records"
        # The smart thing here is the wait: 0 param
        # By default find_all will wait for Capybara.default_max_wait_time as it is waiting for all JS methods 
        # to complete. But there is no JS to complete and we want to check the page as is, without waiting 
        # for any JS, because there is no JS. 
        # 
        # We pase a "wait: 0" which will check and return
        break if find_all(:xpath, "//a[@href='/records/#{record.to_param}' and text()='Continue']", wait: 0).any?
    
        # If we could not find our record we sleep for 0.25 seconds and try again.
        sleep 0.25
      end
    end

    I hope it is helpful.

    Want to keep it touch – find me on LinkedIn or Twitter.

     
  • kmitov 6:41 am on June 5, 2021 Permalink |
    Tags: rails, , , ,   

    Yet another random failing spec 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    This article is about a random failing spec. I spent more than 5 hours on this trying to track it down so I decided to share with our team what has happened and what the stupid mistake was.

    Random failing

    Random failing specs are most of the time passing and sometimes failing. The context of their fail seems to be random.

    Context

    At FLLCasts.com we have categories. There was an error when people were visiting the categories. We receive each and every error on an email and some of the categories stopped working, because of a wrong sql query. After migration from Rails 6.0 to Rails 6.1 some of the queries started working differently mostly because of eager loads and we had to change them.

    The spec

    This is the code of the spec

     scenario "show category content" do
        category = FactoryBot.create(:category, slug: SecureRandom.hex(16))
        episode = FactoryBot.create(:episode, :published_with_thumbnail, title: SecureRandom.hex(16))
        material = FactoryBot.create(:material, :published_with_thumbnail, title: SecureRandom.hex(16))
        program = FactoryBot.create(:program, :published_with_thumbnail, title: SecureRandom.hex(16))
        course = FactoryBot.create(:course, :published_with_thumbnail, title: SecureRandom.hex(16))
    
        category.category_content_refs << FactoryBot.create(:category_content_ref, content: episode, category: category)
        category.category_content_refs << FactoryBot.create(:category_content_ref, content: material, category: category)
        category.category_content_refs << FactoryBot.create(:category_content_ref, content: program, category: category)
        category.category_content_refs << FactoryBot.create(:category_content_ref, content: course, category: category)
    
        expect(category.category_content_refs.count).to eq 4
        visit "/categories/#{category.to_param}"
    
        find_by_xpath_with_page_dump "//a[@href='/tutorials/#{episode.to_param}']"
        find_by_xpath_with_page_dump "//a[@href='/materials/#{material.to_param}']"
        find_by_xpath_with_page_dump "//a[@href='/programs/#{program.to_param}']"
        find_by_xpath_with_page_dump "//a[@href='/courses/#{course.to_param}']"
    
      end

    We add a few objects tot he category and then we check that we see them when we visit the category.

    The problem

    Sometime while running the spec only 1 of the objects in the category are shown. Sometimes non, most of the time all of them are shown.

    The debug process

    The controller

    def show
      @category_content_refs ||= @category.category_content_refs.published
    end

    In the category we just call published to get all the published content that is in this category. There are other things in the show but they are not relevant. We were using apply_scopes, we were using other concerns.

    The model

      scope :published, lambda {
        include_contents.where(PUBLISHED_OR_COMING_WHERE_SQL)
      }

    The scope in the model makes a query for published or coming.

    And the query, i kid you not, that was committed in 2018 and we’ve had this query for so long was

    class CategoryContentRef < ApplicationRecord
       
        PUBLISHED_OR_COMING_WHERE_SQL = ' (category_content_refs.content_type = \'Episode\' AND (episodes.published_at <= ? OR episodes.is_visible = true) ) OR
         (category_content_refs.content_type = \'Course\' AND courses.published_at <= ?) OR
         (category_content_refs.content_type = \'Material\' AND (materials.published_at <= ? OR materials.is_visible = true) ) OR
         category_content_refs.content_type=\'Playlist\'', *[Time.now.utc.strftime("%Y-%m-%d %H:%M:%S")]*4].freeze
    
    end
    

    I will give you a hit that the problem is with this query.

    You can take a moment a try to see where the problem is.

    The query problem

    The problem is with the .freeze and the constant in the class. The query is initialized when the class is loaded. Because of this it takes the time at the moment of loading the class and not the time of the query.

    Because the specs are fast sometimes the time of loading of the class is right before the spec and sometimes there are specs executed in between.

    It seems simple once you see it, but these are the kind of things that you keep missing while debugging. They are right in-front of your eyes and yet again sometimes you just can’t see them, until you finally see them and they you can not unsee them.

     
  • kmitov 3:19 pm on May 31, 2021 Permalink |
    Tags: , rails,   

    When caching is bad and you should not cache. 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    On Friday we did some refactoring at FLLCasts.com. We removed Refinery CMS, which is a topic for another article, but one issue pop-up – on a specific page caching was used in a way that made the page very slow. This article is about how and why. It is mainly for our team as a way to share the knowledge among ourselves, but I think the whole community could benefit, especially the Ruby on Rails community.

    TL;DR;

    When making a request to a cache service, be it MemCachir, Redis or any other, you are making a request to a cache service. This will include a get(key) method call and if the value is not stored in the cache, it will include a set(key) method call. When the calculation you are doing is simple it will take more time to cache the result from the calculation than to do the calculation again, especially if this calculation is a simple string concatenation.

    Processors (CPUs) are really good at string concatenation and could do them in a single digit milliseconds. So if you are about to cache something, make sure that you cache something worth caching. There is absolutely no reason to cache the result of:

    # Simple string concatenation. You calculate the value. No need to cache it.
    value = "<a href=#{link}>Text</a>". 
    
    # The same result, but with caching
    # There isn't a universe in which the code below will be faster than the code above.
    hash = calculate_hash(link)
    cached_value = cache.get(hash)
    if cached_value == nil
       cached_value = "<a href=#{link}>Text</a>". 
       cache.set(hash, cached_value)
    end 
    
    value = cached_value

    Context for Rails

    Rails makes caching painfully easy. Any server side generated HTML could be cached and returned to the user.

    <% # The call below will render the partial "page" for every page and will cache the result %>
    <% # Pretty simple, and yet there is something wrong %>
    <%= render partial: "page", collection: @pages, cached: true %>

    What’s wrong is that we open the browser and it takes more than 15 seconds to load.

    Here is a profile result from New Relic.

    As you can see there a lot of Memcached calls – like 10, and a lot of set calls. There are also a lot of Postgres find methods. All of this is because of how caching was set up in the platform. The whole “page” partial, after a decent amount of refactoring turns out to be a simple string concatenation as:

    <a href="<%= page.path%>"><%= page.title %></a>

    That’s it. We were caching the result of a simple string concatenation which the CPU is quite fast in doing. Because there were a lot of pages and we were doing the call for all of the pages, when opening the browser for the first time it just took too much to call all the get(key), set(key) methods and the page was returning a “Time out”

    Conclusion

    You should absolutely use caching and cache the values of your calculations, but only if those calculations take more time than asking the cache for a value. Otherwise it is just not useful.

     
  • kmitov 9:14 am on May 7, 2021 Permalink |
    Tags: , rails   

    “[DOM] Input elements should have autocomplete attributes” 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    This is one of the things that could make a platform better. Here is how the warning looks like in the browser console.

    More information at – https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/autocomplete

    The autocomplete attributes could allow browsers, extensions and other agents guess what the user should do on this page. It could make it easier for the user. For example an extension could suggest a new password in the field, or could understand to fill the name of the user in the “name” field.

    Additionally we don’t like warnings.

    To check out the behavior, if you have a password manager for example go to

    https://www.fllcasts.com/users/sign_in

    or

    https://www.buildin3d.com/users/sign_in

     
  • kmitov 8:39 am on May 7, 2021 Permalink |
    Tags: csrf, rails,   

    [Rails] Implementing an API token for post requests 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    At the BuildIn3D platform we provide clients with API to send certain HTTP POST requests. Questions is – how do we authenticate them.

    Here is one of the authentication steps – we implemented our own build_token. When authenticity_token for CSRF is available we also use the authenticity_token. But it is not always available because the authenticity_token depends on the session and the session cookie. But there might not be a session and a cookie in some cases and yet we still need some authentication. Here is how we do it.

    Generate a Unique encrypted token on the server side

    The server generates a token based on pass params. This could be username or password or other info.

        def to_build_token
          len   = ActiveSupport::MessageEncryptor.key_len
          salt  = Rails.appplicaton.secret_build_token_salt
          key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
          crypt = ActiveSupport::MessageEncryptor.new(key)
          encrypted_data = crypt.encrypt_and_sign(self.build_id)
          Base64.encode64(encrypted_data)
        end

    This will return a new token that has encrypted the build_id.

    encrypted_data = crypt.encrypt_and_sign(self.build_id)
    # We could easily add more things to encrypt, like user, or some params or anything you need to get back as information from the token when it is later submitted

    Given this token we can pass this token to the client. The token could expire after some time.

    We would require the client to send us this token on every request from now on. In this way we know that the client has authenticated with our server.

    Decryption of the token

    What we are trying to extract is the build_id from the token. The token is encrypted so the user can not know the secret information that is the build_id.

    def self.build_id_from_token token
      len   = ActiveSupport::MessageEncryptor.key_len
      salt  = Rails.application.secret_salt_for_build_token
      key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
      crypt = ActiveSupport::MessageEncryptor.new(key)
      crypt.decrypt_and_verify(Base64.decode64(token))
    end

    Requiring the param in each post request

    When a post request is made we should check that the token is available and it was generated from our server. This is with:

      def create
          build_token = params.require("build_token")
          build_id_from_token = Record.build_id_from_token(build_token)
          .... # other logic that now has the buid_id token
      end

    The build token is one of the things we use with the IS at BuildIn3D and FLLCasts.

    Polar bear approves of our security.

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel