Are you curious about what is inside those phishing emails and how they try to steal your password?
This is the story of what happens when you click on one of the phishing emails that we receive so often. If you’ve ever been curious about how these emails work, and how they look, I will be happy to help without burdening you with tech details.
A couple of days ago our CEO received an email that looked real, but was trying to steal her password for Microsoft Office.
Note: Don’t click on links and email attachments. What I am doing here is demonstrating the content of one of these emails in a controlled sandbox environment.
Content of phishing email
This is the email. It has an attachment. Looks kind of real.
This is how the attachment looks like:
What could you do:
Check the sender
Ask your CTO/Admin/Security or somebody with good technical knowledge whether this email is ligit.
Don’t click on attachments
The attachment is an HTM file
This file is executable on Microsoft Windows machines. Lets see the content of this file if you open it with a text editor:
This file contains an HTML document. HTML is the format of webpages and once you click on this file it will open your web browser and the web browser will execute this file.
Note: Don’t click on such attachments.
What is unescape?
This unescape here means that the string
“‘%0D%0A%20%20%20%20%20%20%20%20%3C%73%63%72%69%70%74%20%73%72%63%3D%27%68%74%74%70%73%3A%2F%2F%63%64..” is encoded.
‘unescape’ is a function that changes the encoding. It is technical, but at the end the goal is to make sure this file could be read by all browsers.
The result of unescaping the content looks like:
<script src='https://cdn.jsdelivr.net/npm/crypto-js@4.1.1/crypto-js.js'></script>
<script src='https://cdn.jsdelivr.net/npm/crypto-js@4.1.1/aes.js'></script>
<script src='https://cdn.jsdelivr.net/npm/crypto-js@4.1.1/pbkdf2.js'></script>
<script src='https://cdn.jsdelivr.net/npm/crypto-js@4.1.1/sha256.js'></script>
<script>
function CryptoJSAesDecrypt(passphrase, encrypted_json_string){
var obj_json = JSON.parse(encrypted_json_string);
var encrypted = obj_json.ciphertext;
var salt = CryptoJS.enc.Hex.parse(obj_json.salt);
var iv = CryptoJS.enc.Hex.parse(obj_json.iv);
var key = CryptoJS.PBKDF2(passphrase, salt, { hasher: CryptoJS.algo.SHA256, keySize: 64/8, iterations: 999});
var decrypted = CryptoJS.AES.decrypt(encrypted, key, { iv: iv});
return decrypted.toString(CryptoJS.enc.Utf8);
}
document.write(CryptoJSAesDecrypt('978421099', '{"ciphertext":"E8jA2IVItrQQ0SW+CsN1+bRVk2bXLpW5OefWqfRyHU0qa6qTVv379y5qP2rlaRmdNkpeHJ+5t+szBF\/V7UyFG\/dxUWfgifts\/HvH38XW0qufGiryCqLxx0oo9YYtg8Qq8N1Wqg4tNiuYsdy\/RAneSerZBDpWTwUtiDE6rx6yhRNaYpRMxsUODzToXEoGWfcoFSiSAUY3mA2rhDSNeSe9WxnrMlGxRJ5VedyYDdqz8aQ24s\/Y+nIwE
Here is what happens in the code in simple terms:
There is an encrypted text called “ciphertext” and this cipher text is decrypted and execute. This happens on the last line of the fragment above.
So the phishing main contains an attachment, this attachment is ‘escaped’ and the ‘escaped’ content is encrypted.
What’s the content of the ciphertext?
The cipher text contains a web page that you browser will visualize. It looks like a real web page. It looks like a real Microsoft 365 page.
Here is a screenshot:
Here where you see “pesho@gmail.com” you will see your personal email. This makes the page look more real to you.
The summary for now – the phishing email contains an attachment, that has an executable HTML code that is escaped, that is also encrypted, and the encrypted content contains an HTML page that looks like Microsoft login page.
What happens when you fill your email and password?
There is a fragment of the code of the page that looks like this:
So I will not even understand that I was compromised.
What can you do?
Add a two factor authentication.
That’s the easiest, most secure solution. Add a two factor authentication that will send you an SMS every time you login or will require you to use an authenticator app.
If you haven’t done it already, I would advice you to do it now.
On the Internet you are nobody until someone attacks you.
It gets even more interesting when the attack comes from someone with practically unlimited resources and when these are the same people that are supposed to protect you.
This article is the story of how Cloudflare started an “attack” on a machine at the FLLCasts platform. This increased the traffic of the machine about 10x and AWS started charging the account 10x more. I managed to stop them and I hope my experience is useful for all CTOs, sysadmins, devops and others that would like to understand more and look out for such cases.
TL; DR;
Current up to date status is – after all the investigation it turns out that when a client makes a HEAD request for a file this will hit Cloudflare infrastructure. Cloudflare will then send a GET request to the account machine and will cache the file. This has changed at 28 of August. Before 28 of August when clients were sending HEAD requests, Cloudflare was sending HEAD requests (that don’t generate traffic). After 28 of August clients are still sending HEAD requests, but now Cloudflare is sending GET requests, generating terabytes of additional traffic that is not needed.
Increase of the Bill
On 28 of August 2021 I got a notification from AWS that the account is close to surpassing its budget for the month. This is not surprising as it was the end of the month, but nevertheless I decided to check. It seems that the traffic to one of the machines has increased 10x in a day. Nothing else has increased. No visits, no other resources, just the traffic to this one particular machine. That was strange. This has been going on for 7 days now and this is the increase of the traffic.
AWS increase of the bill
Limit billing on AWS
First thought was “How can I set a global limit to AWS spending for this account? I don’t want to wake up with $50K in traffic charges the next day?”
The answer is “You can’t”. There is no way to set a global spending limit for an AWS account. This was something I already knew, but decided to check again with support and yes, you can’t set such a limit. This means that AWS is providing all the tools for you to be bankrupt by a third party and they are not willing to limit it.
Limit billing on Digital Ocean
I have some machines on Digital Ocean and I checked there. “Can I set a global spending limit for my account where I will no longer be charged and all my services will stop if my spending is above X amount of dollars?”. The answer was again – “No. Digital ocean does not provide it”.
Should there be a global limit on spending on cloud providers?
My understanding is – yes. There is a break even point where users are coming to your service and generating revenue and you are delivering the service and this is costing you money. Once it costs you more to deliver the service than the revenue that the service is generating, I would personally prefer to stop the service. No need for it to be running. Otherwize you could wake up with a $50K bill.
AWS monitoring
I had the bill from AWS so I tried looking at the monitoring. There is a spike every day between 03:00 AM UTC and 05:00 AM UTC. This spike is increasing the traffic with hundreds of gigabytes. It could easily be terabytes next time. The conclusion is that the machine is heavily loaded during this time.
AWS monitoring
Nginx access.log
Looking at the access log I see that there are a lot of requests by machines that are using a user agent called ‘curl’. ‘curl’ is a popular tool for accessing files over HTTP and is heavily used by different bots. But bots tend to identify themselves.
The result is 547 machines. The full log file is available at – Full list of Cloudflare IPs attacking my machine. The top 20 are (there are some IPs that are not from Cloudflare). The first is the number of requests, the second is the IP of the machine.
Looking at the machines that are making the requests these are 547 different machines, most of which are Cloudflare machines. These are servers that Cloudflare seems to be running that are making the request.
How does Cloudflare work?
For this particular FLLCasts account with this particular machine I have years ago setup Cloudflare to sit in front of the machine to help protect the account from internet attacks.
The way Cloudflare works is that only Cloudflare knows what is the IP address of our machine. This is the promise that Cloudflare is making. Because only they know the IP address of the machine, only they know what is the IP address for a given domain. In this way when a user points their browser at the “http://domainname” the internet will direct this request to Cloudflare, then Cloudflare will check if this request is ok, and then and only then forward this request to our machine. But in the meantime Cloudflare is trying to help businesses like the platform by caching the content. This means that when Cloudflare receives a request for a file, they will check on their Cloudflare infrastructure if this file was cached and send a request to the account machine only if there is no cache.
In a nutshell Cloudflare maintains a cache for the content the platform is delivering.
Cloudflare maintains a cache of the platform resources. Every night between 03:00 AM UTC and 05:00 AM UTC some 547 Cloudflare machines decide to update their cache and they start sending requests to our server. These are 10x more requests that the machine generally receives from all users. The content on the server does not change. It’s been the same content for years. But for the last 7 days Cloudflare is caching the same content every night on 547 machines.
And AWS bills us for this.
Can Cloudflare help?
I created a ticket. The response was along the lines of “You are not subscribed for our support, you can get only community support”. Fine. I called them on the phone early in the morning. I called enterprise sales and I asked them.
Me - "Hi, I am under attack. Can you help?"
They - "Yes, we can help. Who is attacking you?"
Me - "Well, you are. Is there an enterprise package I could buy so that you can protect me against your attack?"
Luckily the guy on the phone caught my sense of humor and urgency and quickly organized a meeting with a product representative. Regrettably there were no solution engineers on this call.
Both guys were very knowledgeable, but I had difficulties explaining that it was actually Cloudflare causing the traffic increase. I had all the data from AWS, from the access.log files, but the support agents still had some difficulty accepting it.
To be clear – I don’t think that Cloudflare is maliciously causing this. There is no point. What I think has happened is some misconfiguration on their side that caused this for the last 7 days.
What I think has happened?
I tried to explain to the support agents that there are three scenarios all of which Cloudflare is responsible for.
1. Option 1 – “someone that has 547 machines is trying to attack the FLLCasts account and Cloudflare is failing to stop it”. First this is very unlikely. Nobody will invest in starting 547 machines just to make the platform pay a few dollars more this month. And even if this is the case, this is what Cloudflare should actually prevent, right? Option 1: “Cloudflare is failing in preventing attacks” (unlikely)
2. Option 2 – “only Cloudflare knows the IP of this domain name and they have been compromised.”. The connection between domain name and ip address is something that only Cloudflare knows about. If a third party knows the domain name and they are able to find the IP name this means that they are compromising Cloudflare. Option 2: “Cloudflare is compromised” (possible, but again, unlikely)
3. Option 3 – “there is a misconfiguration in some of the Cloudflare servers”. I don’t like looking for malicious activity where everything could be explained with simple ignorance or a mistake. Most likely there is a misconfiguration in the Cloudflare infrastructure that is causing these servers to behave in this way. Option 3: “There is a misconfiguration in Cloudflare infrastructure”
4. Option 4 – “there is a mistake on our end”. As there basically is nothing on our end and this nothing has not changed in years, the possibility for this to be the case is minimal.
On a support call we set a plan with the support agents to investigate it. I will change the public IP of the AWS machine and will reconfigure it on Cloudflare. In this way we hope to stop some of the requests. We have no plan for what to do after that.
Can I block in on the Nginx level?
Nginx is an HTTP server,serving files. There are a couple of options to explore there, but the most reasonable was to stop all curl requests to the Nginx server. This was the shortest path. There was no need to protect against other attacks, there was only the need to protect against Cloudflare attacks. The Cloudflare attack was using “curl” as a tool. I decided to stop ‘curl’
# Surely not the best, but the simplest and will get the job done for now.
if ($http_user_agent ~ 'curl') {
return 444; # Consider returning 444. It's a custom nginx code that drop the connection without responding.
}
Resolution
I am now waiting to see if the change of the public IP of the AWS machine will have any impact and if not I am just rejecting all “curl” requests that seem to be what Cloudflare is using.
Update 1
The first solution that we decide to implement is to
Change the public IP of the AWS machine and change it in the DNS settings at Cloudflare. In this way we would make sure that only Cloud flare really knows this IP.
Resolution is – It did not work!
I know it won’t, because it was another way for support to get me to do anything without really looking into the issue, but I went along with it. Better exhaust this options and be sure.
The traffic of a Cloudflare attacked machine. Changing the IP address of 03 of September had no effect.
Update 2
Adding CF-Connection-IP header
Cloudflare support was really helpful. They asked me to include CF-Connection-IP in the logs. In this way we would know what is the real IP that is making the requests and if these are in fact Cloudflare machines.
Cloudflare is making GET when client makes a HEAD request
This is what I found out. The platform has a daily job that checks the machine and makes sure files are ok. This integrity check was left there from times when we had to do it, like years ago. It is still running and is starting every night checking the machine with HEAD requests. But Cloudflare started making GET request at 28 of August 2021 and this increases the traffic to the machine.
Steps to reproduce
Here are the steps to reproduce:
1. I am sending a HEAD request with ‘curl -I’
2. Cloudflare has not cached the file so there is “cf-cache-status: MISS”
3. Cloudflare sends a GET request and gets the whole file
4. Cloudflare responds to the HEAD request.
5. I send a HEAD request agian with ‘curl -I’
6. Cloudflare has the file cached and there is a “cf-cache-status: HIT”
7. The account server is not hit.
The problem here is that I am sending a HEAD request to my file and Cloudflare is sending a GET request for the whole file in order to cache this file
Note that for the last HEAD request there is a “cf-cache-status: HIT”.
Status and how it could be resolved?
Yes, we are doing HEAD requests every day to the files in order to check that they are all working. Every day we send a HEAD request for every file to make sure all files are up to date. This has been going on for years and is a left over of an integrity check we implemented in 2015.
What has changed on 28 of August 2021 is that when Cloudflare receives a HEAD request for a file it is sending a GET request to our machine in order to cache the file. This is what has changed and this is generating all the traffic.
We send HEAD request with ‘curl -I’
I have 30 weeks of log files that show that Cloudflare was sending HEAD requests like
I have asked Cloudflare
Could you please rollback this change in the infrastructure and do not send a GET request to our machine when you receive a HEAD request from a client?
Let’s see how will this be resolved.
Up to date conclusion
Check your machines from time to time. I hope you don’t get in this situation.
Want to keep in touch? – find me on LinkedIn or Twitter
(Everyday Code – instead of keeping our knowledge in a README.md let’s share it with the internet)
At the BuildIn3D platform we provide clients with API to send certain HTTP POST requests. Questions is – how do we authenticate them.
Here is one of the authentication steps – we implemented our own build_token. When authenticity_token for CSRF is available we also use the authenticity_token. But it is not always available because the authenticity_token depends on the session and the session cookie. But there might not be a session and a cookie in some cases and yet we still need some authentication. Here is how we do it.
Generate a Unique encrypted token on the server side
The server generates a token based on pass params. This could be username or password or other info.
def to_build_token
len = ActiveSupport::MessageEncryptor.key_len
salt = Rails.appplicaton.secret_build_token_salt
key = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
crypt = ActiveSupport::MessageEncryptor.new(key)
encrypted_data = crypt.encrypt_and_sign(self.build_id)
Base64.encode64(encrypted_data)
end
This will return a new token that has encrypted the build_id.
encrypted_data = crypt.encrypt_and_sign(self.build_id)
# We could easily add more things to encrypt, like user, or some params or anything you need to get back as information from the token when it is later submitted
Given this token we can pass this token to the client. The token could expire after some time.
We would require the client to send us this token on every request from now on. In this way we know that the client has authenticated with our server.
Decryption of the token
What we are trying to extract is the build_id from the token. The token is encrypted so the user can not know the secret information that is the build_id.
def self.build_id_from_token token
len = ActiveSupport::MessageEncryptor.key_len
salt = Rails.application.secret_salt_for_build_token
key = ActiveSupport::KeyGenerator.new(Rails.application.secret_key_base).generate_key(salt, len)
crypt = ActiveSupport::MessageEncryptor.new(key)
crypt.decrypt_and_verify(Base64.decode64(token))
end
Requiring the param in each post request
When a post request is made we should check that the token is available and it was generated from our server. This is with:
def create
build_token = params.require("build_token")
build_id_from_token = Record.build_id_from_token(build_token)
.... # other logic that now has the buid_id token
end
The build token is one of the things we use with the IS at BuildIn3D and FLLCasts.
Reply
You must be logged in to post a comment.