Updates from October, 2021 Toggle Comment Threads | Keyboard Shortcuts

  • kmitov 9:00 am on October 5, 2021 Permalink |
    Tags:   

    A day of the life of a CTO – “what do you do, day to day?” 

    My brother asked me the other day:

    He: – So what do you do, day to day?

    Me: – I work in engineering (software, data and AI).

    It’s a little bit more than that. Not all the days are the same. There are a lot of decisions to be made, and generally with a little luck those decisions will keep the ship at least in the right direction.

    I decided to look deeper and to record. There is a difference between what we think we are doing and what we are actually doing. I tried to summarize just one of my recent days that I spent in engineering. This was a day without any software development for me.

    My hope with this article is to be able to answer my brother – “What do you do, day to day?” and I hope this answer and examples could be interesting to people entering the world of Software engineering and to business and product people trying to learn more about how their engineers spend their day.

    Adding a JSONB column to a scheme

    A colleague was facing the issue of storing an array of values in a DB. The values were the result of calling the API of an external service for our business.

    How do you store these values? There are many different ways. I supported his recommendation to store the data as JSON format. I only suggested changing the type of the column to JSONB as this will later allow us to query the table in an easier way. At the same time I had to re-think part of the stack to see if there would be any implications on the whole platform when this new column JSONB is introduced. Luckily there were no implications.

    Automated test that we create a table

    A colleague was working on a pipeline and the specs for this pipeline. The question was how do we build a spec for part of the logic. The logic creates a new table. How do we test this in an automated way? How do we test that this logic creates a db? We considered two different approaches and together we looked for a good API call to test this.

    We decided on how to spec this behavior.

    DateTime field

    A colleague was facing the issue with date in the data platform that had invalid values. The issue was that we were storing both the date and the time for an event where we should have been storing only the time. The implications were huge. We now had to migrate all the values in all the records.

    In this case we looked at the product requirements on what should be included in the data platform in the near future. Turns out that there is a requirement for engineering to store not only the time, but also the date of the event. This means there was nothing to fix as we were ahead of time in engineering. We only have to migrate a couple of records.

    The decision here was whether we should spend a day migrating this data and what would be the issue if the data was not migrated.

    Choose a service $50/80GB

    A colleague had the task to look at different services that we could use. We had to make a decision should we use this $50 service or that $50 service. The decision is important because once you decide on a service to add in your stack it is difficult to move out of this service. You kind of stay with them at least for the near future.

    Sometimes when you look at two services on your own you can overlook a few of the aspects so it is a good practice to have a second look from someone else. Also at the end it is a team decision of what to include in the stack.

    Integration with an external API

    A colleague was working on integrating with an external API. The issue was the this API is returning different formats for different calls. The question was how do we handle this. Should we hard code the scheme for this API, should we infer it, should we do something smarter? How does this impact the abstraction for the other Data Sources. We had to get on a call with the external API representatives to discuss if they could help us.

    Creating new repos

    A colleague was working on new features in the platform. These new feature should be extracted into new repositories. We had to decide on the name of the repositories. In the world of software development there are two hard things – invalidating cache and naming. Naming is important because it gives you power over things. Once you name them you have power over them. If you name them bad, then they have power over you. Nevertheless, we had to make a decision on how we name two new code repositories.

    Abilities

    A colleague was working on the authorization part of the platform. We are adding new authorizations based on roles. He developed the code and was ready for a Code review. I was there and decided to jump on the Code Review. The issue with the implementation was that it was coupling the authorization with all the modules in a single class. Coupling is bad in the long run as it is not very agile and difficult to maintain. We spent time decoupling the implementation.

    System vs model specs

    A colleague was in the middle of developing an automated spec. There are generally two types of specs – integration and unit. In our case we use “system” and “model” specs. System specs test the behavior of the whole feature. Unit specs test the behavior of a specific unit (class, function). My general rule of thumb is – 10% system specs, 90% model specs, but start with the system spec. I’ve been in situations with too many system specs which make the system unmaintainable and require us to invest a lot of time in refactoring. Since then I am cautious about what kind of spec are developed, when and why. We revised current assumptions and decided if current specs should be developed as a system or unit.

    Flash messages

    A colleague was working on some flash messages on the platform that are appearing at a specific moment. I took a look and confirmed the implementation and the behavior.

    Constructing new objects

    A colleague was working on refactoring part of the code. A general rule of thumb I try to follow is “always construct instances of a given type” only at one specific place in the code. We revised the implementation and saw that there are a few places where instances of a given type were constructed. There is an easy solution for this. We schedule it for the following week to be implemented.

    Submit button change type to input

    A colleague was working on a feature on the web platform and noticed that a few of the forms had the wrong type of button. I was around and I was the one to previously commit this form so he notified me about the change and we discussed the implications.

    Structure of blob storage

    A colleague was working on an integration with an API that will store information in our BigData Lake. We had to sync the structure of the lake and how it will accommodate the new API.

    Infrastructure from code

    A colleague was working on deploying on a cloud provider. We try to create our infrastructure from code. It is way too easy to set up an infrastructure, spend a week deploying it, and then be unable to reproduce it later on because of the gazillion different options and little configurations you have to do on the cloud providers. Just ask anyone who has configured AWS IAM permissions and resources.

    It is important to have a script that would create your infrastructure from code. We had to revise, review and think about the implications of the resources that the code creates.

    Conclusion

    No actual conclusion. This is just a diary of the day. I hope that my brother along with many others now understand more about our work.

     

  • kmitov 4:30 am on September 8, 2021 Permalink |
    Tags: airflow, apache, bigdata   

    Orchestration of BigData with Apache Airflow 

    It was a please for me to do this presentation and discuss how we can orchestrate BigData with Apache Airflow at the 2021 OpenFest event

    Video is in Bulgarian

     

  • kmitov 9:48 am on August 27, 2021 Permalink
    Tags: customer,   

    How we lost $1000 because we did not talk to the customer early enough 

    This content is password protected. To view it please enter your password below:

     

  • kmitov 4:19 pm on June 13, 2021 Permalink |
    Tags: , , ,   

    Dependencies – one more variable adding to the “cost of the code” 

    One thing I have to explain a lot is what are the costs of software development. Why are things taking so long? Why is there any needed for maintenance and support? Why are developers spending significant amount of their time looking over the existing code base and why we can not just add the next and the next feature?

    Today I have an example of this – and these are “dependencies”.

    The goal of this article is to give people more understanding on how the “tech works.”. I’ve seen that every line of code and every dependency that we add to a project will inevitably result in further costs down the road so we should really keep free of unnecessary dependencies and features.

    Daily builds

    Many contemporary professional software projects have a daily build. This means that every day at least once the project is “built” from zero, all the tests are run and we automatically validate that the customers could use it.

    Weekly dependencies updates

    Every software project depends on libraries that implement common functionality and features. Having few dependencies is healthy for the project, but having no dependencies and implementing everything on your own is just not viable in today’s world.

    These libraries and frameworks that we depend on also regularly release new versions.

    My general rule that I follow in every project is that we check for new versions of the dependencies every Wednesday at around 08:00 in the morning. We check for new dependencies, we download them, we build the project and we run the specs/tests. If the tests fail this means that the new dependencies that we’ve downloaded have somehow changed the behavior of the project.

    Dependencies change

    Most of the time dependencies are changed in a way that does not break any of the functionality of your project. This week was not such a week. A new dependency came along and it broke a few of the projects.

    The problem came from a change in two dependencies:

    Fetching websocket-driver 0.7.5 (was 0.7.4)
Fetching mustache-js-rails 4.2.0.1 (was 4.1.0)
Installing mustache-js-rails 4.2.0.1 (was 4.1.0)
Installing websocket-driver 0.7.5 (was 0.7.4) with native extensions

    We have installed new versions of two of the dependencies “websocket-driver” and “mustache-js-rails’

    These two dependencies broke the builds.

    Why should we keep up to date

    Now out of the blue we should resolve this problem. This takes time. Sometimes it is 5 minutes. Sometimes it could be an hour or two. If we don’t do it, it will probably result in more time at a later stage. As the change is new in ‘mustache-js-rails’ we have the chance to get in touch with the developers of the library and resolve the issue while it is fresh for them and they are still “in the context” of what they were doing.

    Given the large number of dependencies that each software project has there is a constant need to keep up to date with new recent versions of your dependencies.

    What if we don’t keep up to date?

    I have one such platform. We decided 6-7 years ago not to invest any further in it. It is still working but it is completely out of date. Any new development will cost the same as basically developing the platform as brand new. That’s the drawback of not keeping up to date. And it happens even with larger systems on a state level with the famous search for COBOL developers because a state did not invest in keeping their platform up to date for some 30+ years.

     

  • kmitov 3:19 pm on May 31, 2021 Permalink |
    Tags: , ,   

    When caching is bad and you should not cache. 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    On Friday we did some refactoring at FLLCasts.com. We removed Refinery CMS, which is a topic for another article, but one issue pop-up – on a specific page caching was used in a way that made the page very slow. This article is about how and why. It is mainly for our team as a way to share the knowledge among ourselves, but I think the whole community could benefit, especially the Ruby on Rails community.

    TL;DR;

    When making a request to a cache service, be it MemCachir, Redis or any other, you are making a request to a cache service. This will include a get(key) method call and if the value is not stored in the cache, it will include a set(key) method call. When the calculation you are doing is simple it will take more time to cache the result from the calculation than to do the calculation again, especially if this calculation is a simple string concatenation.

    Processors (CPUs) are really good at string concatenation and could do them in a single digit milliseconds. So if you are about to cache something, make sure that you cache something worth caching. There is absolutely no reason to cache the result of: 

    # Simple string concatenation. You calculate the value. No need to cache it.
value = "<a href=#{link}>Text</a>". 

# The same result, but with caching
# There isn't a universe in which the code below will be faster than the code above.
hash = calculate_hash(link)
cached_value = cache.get(hash)
if cached_value == nil
   cached_value = "<a href=#{link}>Text</a>". 
   cache.set(hash, cached_value)
end 

value = cached_value

    Context for Rails

    Rails makes caching painfully easy. Any server side generated HTML could be cached and returned to the user. 

    <% # The call below will render the partial "page" for every page and will cache the result %>
<% # Pretty simple, and yet there is something wrong %>
<%= render partial: "page", collection: @pages, cached: true %>

    What’s wrong is that we open the browser and it takes more than 15 seconds to load.

    Here is a profile result from New Relic.

    As you can see there a lot of Memcached calls – like 10, and a lot of set calls. There are also a lot of Postgres find methods. All of this is because of how caching was set up in the platform. The whole “page” partial, after a decent amount of refactoring turns out to be a simple string concatenation as:

    <a href="<%= page.path%>"><%= page.title %></a>

    That’s it. We were caching the result of a simple string concatenation which the CPU is quite fast in doing. Because there were a lot of pages and we were doing the call for all of the pages, when opening the browser for the first time it just took too much to call all the get(key), set(key) methods and the page was returning a “Time out”

    Conclusion

    You should absolutely use caching and cache the values of your calculations, but only if those calculations take more time than asking the cache for a value. Otherwise it is just not useful.

     

  • kmitov 9:14 am on May 7, 2021 Permalink |
    Tags: ,   

    “[DOM] Input elements should have autocomplete attributes” 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    This is one of the things that could make a platform better. Here is how the warning looks like in the browser console.

    More information at – https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/autocomplete

    The autocomplete attributes could allow browsers, extensions and other agents guess what the user should do on this page. It could make it easier for the user. For example an extension could suggest a new password in the field, or could understand to fill the name of the user in the “name” field.

    Additionally we don’t like warnings.

    To check out the behavior, if you have a password manager for example go to

    https://www.fllcasts.com/users/sign_in

    or

    https://www.buildin3d.com/users/sign_in

     

  • kmitov 8:39 am on May 7, 2021 Permalink |
    Tags: csrf, ,   

    [Rails] Implementing an API token for post requests 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    At the BuildIn3D platform we provide clients with API to send certain HTTP POST requests. Questions is – how do we authenticate them.

    Here is one of the authentication steps – we implemented our own build_token. When authenticity_token for CSRF is available we also use the authenticity_token. But it is not always available because the authenticity_token depends on the session and the session cookie. But there might not be a session and a cookie in some cases and yet we still need some authentication. Here is how we do it.

    Generate a Unique encrypted token on the server side

    The server generates a token based on pass params. This could be username or password or other info. 

        def to_build_token
      len   = ActiveSupport::MessageEncryptor.key_len
      salt  = Rails.appplicaton.secret_build_token_salt
      key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
      crypt = ActiveSupport::MessageEncryptor.new(key)
      encrypted_data = crypt.encrypt_and_sign(self.build_id)
      Base64.encode64(encrypted_data)
    end

    This will return a new token that has encrypted the build_id. 

    encrypted_data = crypt.encrypt_and_sign(self.build_id)
# We could easily add more things to encrypt, like user, or some params or anything you need to get back as information from the token when it is later submitted

    Given this token we can pass this token to the client. The token could expire after some time.

    We would require the client to send us this token on every request from now on. In this way we know that the client has authenticated with our server.

    Decryption of the token

    What we are trying to extract is the build_id from the token. The token is encrypted so the user can not know the secret information that is the build_id. 

    def self.build_id_from_token token
  len   = ActiveSupport::MessageEncryptor.key_len
  salt  = Rails.application.secret_salt_for_build_token
  key   = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
  crypt = ActiveSupport::MessageEncryptor.new(key)
  crypt.decrypt_and_verify(Base64.decode64(token))
end

    Requiring the param in each post request

    When a post request is made we should check that the token is available and it was generated from our server. This is with:

      def create
      build_token = params.require("build_token")
      build_id_from_token = Record.build_id_from_token(build_token)
      .... # other logic that now has the buid_id token
  end

    The build token is one of the things we use with the IS at BuildIn3D and FLLCasts.

    Polar bear approves of our security.

     

  • kmitov 8:05 am on May 7, 2021 Permalink |
    Tags: ,   

    [Rails] Disabling Forgery Protection on API controllers 

    Forgery protection comes for free in Ruby on Rails and is described in the security guide – https://guides.rubyonrails.org/security.html#cross-site-request-forgery-csrf.

    You don’t want forgery protection on some API controllers. If the API controllers extend the ActionController::Base the forgery protection could be disabled in the following way. Here is an example for a controller accepting requests from Amazon AWS SNS.

    class SnsController < ActionController::Base

  # Disable it for this controller. 
  # If there is no session it is just null session
  protect_from_forgery with: :null_session

  http_basic_authenticate_with :name => ENV["SNS_USERNAME"], :password => ENV["SNS_PASSWORD"]

  ...
end

    An even better approach would be not extending from ActionController::Base, but from ActionController::API. But then we would have to include the modules for HttpAuthentication which is a topic for another article.

     

  • kmitov 7:56 am on May 7, 2021 Permalink |
    Tags: ,   

    [Rails] Please use symbols for polymorphic route arguments 

    This error occurred today with our platform. Issue is at https://github.com/rails/rails/issues/42157

    The issue occurs because of a new security patch with with Rails tracked with CVE-2021-22885.

    You can no longer call polymorphic_path or other dynamic path helpers with strings, because if this strings are provided by the user this could result in unwanted router helper calls.

    # previous call
polymorphic_path([article, "authors"])
# should now be
polymorphic_path([article, "authors".to_sym])
# or better
polymorphic_path([article, :authors])

    Migration effort

    A perspective on how serious it is to upgrade – tl;dr – it is not – about half an hour.

    All the calls in our platform for polymorphic_path

    $ git grep  "polymorphic_path" | wc -l
321

    All the file that have calls to polymorphic_path

    $ git grep -l  "polymorphic_path" | wc -l
143

    Numbers of files that I’ve changed – 13
    Time it took me to review all of them -16 minutes, from 18:24 to 18:40

    Now I am waiting for specs to pass.

    I guess it is not a big change and could be migrate in half an hour. Most of our calls were using symbols already and only about 15 calls from 321 were with strings. These are 4% of all the calls.

     

  • kmitov 5:03 am on April 14, 2021 Permalink |
    Tags: , ,   

    Rendering plain text from the server is not plain text in the browser 

    (Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)

    One thing that is good to know:

    If you render plain text from a web framework like Rails you will not have plain text in the browser. You will still have HTML in the browser.

    This bite us with one of the RSpec scenarios that we were running and I thought I should share with the team and the whole community.

    IRL

    The real life scenario is the following.

    In the Rails controller we do 

    class DigestMessagesController < ApplicationController
    def show
        render plain: "Text"
    end
end

    What we would expect to see in the browser is a simple “Text” text. If fact this is exactly what is returned by the server – you can see in the “Response” tab.

    you can see that the response from the server is just “Text”

    But if you now Inspect the content of the page in the browser you will see that there is HTML inside.

    
<html>
 <head></head>
 <body>
 <pre style="word-wrap: break-word; white-space: pre-wrap;">Text</pre>
 </body>
</html>

    This HTML was not returned by the framework. This HTML was added by the browser. This is not good because an RSpec system scenario with Capybara will pass with rack_test driver, but will fail with a Selenium driver. rack_test driver will not add this HTML while all the major browsers will add it.

    scenario 'for text' do
      visit "/digest_messages/#{digest_message.id}?preview=plain"
      expect(page.body.to_s).to include("Text")
      # This expect here will fail for Selenium and will pass for rack_test driver.
      expect(page.body.to_s).not_to include("html")
end

    I hope this is helpful and could save you a couple of minutes debugging.

     

← Older posts

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel