Personal Project

Monday, June 24, 2019

How to let JavaScript call Native functions in WebView on iOS?

You may think about using an open source framework to bridge the native app with JavaScript. In fact, it is not necessary as long as you know what Apple's framework feature can provide for us. It's very simple to implement without using any heavy open source framework.


Below show 4 solutions on how to create JavaScripts to call native app in WebView for iOS.


URL interception:

  • must redirect or refresh the page, so as the app can catch the redirect URL in function shouldStartLoadWithRequest
  • Supports iOS version ranging from 2 to 12
  • Supports UIWebView
  • Framework: UIWebView

This solution is widely used in lots of open source frameworks and compatible with iOS and Android. Android also can use the same method. 


On WebView, we can simply refresh the page by invoking the following JS function 

location.href=[any link]://[any parameters] 

Then the app will intercept and parse this GET request with URL:[any link]://[any parameters] in function shouldStartLoadWithRequest.


WKScriptMessageHandler:

  • Without redirect a page, JS can call the native function directly
  • Supports iOS 8+
  • Supports WKWebView
  • Framework: WebKit
On WebView, the page can directly call the native function by trigger a button or an event. The app can get and parse the event in WKScriptMessageHandler. 


JavaScriptCore:
  • Without redirect a page, JS can call the native function
  • Supports iOS 7+
  • Supports UIWebView, not support WKWebView 
  • Framework: JavaScriptCore
This solution is limited to UIWebView and as same as WKWebView that can let the page directly
call the native function by triggering a button or an event. 

Method interception:

  • Without redirect, but a user needs to interact with the page to show an alert popup and click button
  • Supports iOS 8+
  • Supports WkWebView
  • Framework: WebKit

Thursday, May 25, 2017

Solutions to Solve Writing Contention Problems in Google Datastore

Google Cloud Platform provides lots of technologies to save developer`s time while thinking about how to scale or maintain our services to serve millions of thousands of users. If you are familiar with MYSQL`s architecture, you will face different kinds of problems to serve a high volume of users. The reason is MYSQL implements the blocking mode when writing data to databases. Google Datastore solved this issue and can write a lot of data in parallel. Especially, Google Datastore is designed for high availability and scalability. 

However, you must test and know deeply about how to write simultaneous data in datastore. Otherwise, your services could only serve current 5 requests per second. It seems like a buggy solution. What is worst, Google's documentations do not give you an easy-to-understand example to implement Sharding Counter. You may wonder how to apply this technique to your entity model and how to avoid this common mistake.


Limitations of Google Datastore
  • Any entity group can only be written at rate 1 request per second.
  • If using @ndb.transactional or @ndb.transactional(xg=true) to write the data, your API can only serve current 5 requests per second. Otherwise, you will get an error of writing contention in datastore.

Why is writing data in Datastore so slow?

Because Datastore needs to copy your data globally and make your data with high availability.



Solutions to Solve Writing Contention Problems

  • Sharding Counter 
  • Use Memcache to batch writing requests and do all the operations in memory and return back to your clients
  • Defer a task queue to write data in datastore


In fact, Sharding Counter is just an example provided by Google. The key point is we can do sharding on our entity group with a unique id as shown below. 

The following codes show how to simultaneously write a thousand of Friendship entities in parallel. If we need to improve its performance, just increase the number of NUM_SHARDS.


NUM_SHARDS = 1000
shard_string_index = str(random.randint(0, NUM_SHARDS - 1))
FriendShip(id=shard_string_index,
           user_key='user Id', 
           friend_key='frind Id')


If you have lots of data models needed to update in one request, please use the task queue to update and return back to your clients only a few amount of information.

If you need to write transactional data using @ndb.transactional or @ndb.transactional(xg=true), defer a task queue to get it done and return a few amount of information to the clients.

Depending on your data model's design, you can use Sharding Counter or Memcache or Task Queue or a hybrid approach to performing the best performance using Google's Datastore and Google App Engine.



Monday, May 1, 2017

How to make TURN Server for high availability?

If you want to keep your WebRTC video streaming services online without any downtime, you must pay attention to the availability of TURN Server. Because TURN Server plays an important to help two parties to connect to each other with Video or Audio streaming in different NAT networks.

The following instructions show how to automatically monitor your TURM server and restart it during the downtime.


1. Install pexpect lib in Python 

sudo pip install pexpect --upgrade



2. Edit MonitorStun.py 
- Telnet your TURN Serer 
- If it is down, ssh to your server and restart it  

#!/usr/bin/env python
import socket
import subprocess
import sys
from datetime import datetime
from pexpect import pxssh


# SSH TO TURN SERVER and restart it
def connect_turn_server():
  s = pxssh.pxssh()

  if not s.login ('TURN Server IP', 'SERVER PORT', 'ACCOUNT', 'PASSWORD'):
    print "SSH session failed on login."
    print str(s)
  else:
    print "SSH session login TURN successful"
    s.sendline ('sudo turnserver -c /usr/local/etc/turnserver.conf --daemo')
    s.prompt()         # match the prompt
    print s.before     # print everything before the prompt.
    s.logout()


# Telnet TURN Server to check it is alive or not on PORT 3478 or 3479
# Clear the screen
subprocess.call('clear', shell=True)

# Ask for input
remoteServer    = 'SERVER IP'
remoteServerIP  = socket.gethostbyname(remoteServer)

# Print a nice banner with information on which host we are about to scan
print "-" * 60
print "Please wait, scanning remote host", remoteServerIP
print "-" * 60

# Check what time the scan started
t1 = datetime.now()

# Using the range function to specify ports (here it will scans all ports between 1 and 1024)

# We also put in some error handling for catching errors

try:
    for port in range(3478,3479):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        result = sock.connect_ex((remoteServerIP, port))
        if result == 0:
            print "Port {}:      Open".format(port)
        else:
             print "TURN Server is down"
             connect_turn_server()
             print "restart TURN Server OK"
        sock.close()


except KeyboardInterrupt:
 print "You pressed Ctrl+C"
    sys.exit()

except socket.error:
    print "Couldn't connect to server"
    sys.exit()

                                        
3. Add MonitorStun.py to con job to check TURN Server in every 1 min.


*/1 * * * * /your_path/monitorStun.py

Of course, you can apply this technique to monitor any services such as SIP Proxy with port 5060, Apache with port 80, or Tomcat with port 8080.



Tuesday, November 22, 2016

How to setup a load balancing and failover back-end RDS for WordPress ?

WordPress is widely used to develop webs for multiple platforms. 
It is necessary to set up WordPress to deploy on a load balancing and failover backend RDS if you want to provide a service without any downtime for your customers.

How to get this  done ?

1. Create a Master and Slave database on RDS.

2.Import HyperDB Plugin into WordPress folder.

HyperDB can be download at below.
https://wordpress.org/plugins/hyperdb/


3. Setup the Master and Salve URL at HyperDB`s config file.


4. Performance Test 

The following info shows the test result on AWS.

EC2         Current session                RDS                                           
t2.micro            110 / s                     single master        
t2.medium         210 /s                       one master and one slave    


To conclude, using WordPress framework to develop the website can achieve a high performance at 210 current requests per second on EC2 with t2.medium spec; with t2.micro of EC2, the best throughput can also achieve 110 current requests per second. Thus, I think WordPress is good enough to build a high-performance website by PHP on AWS.


Tuesday, October 4, 2016

Setting up Apache Server with SSL

//Enable the module by typing:

sudo a2enmod ssl

//  Configure SSL
sudo vim /etc/apache2/sites-available/default-ssl.conf

   ServerAdmin your gmail
   ServerName weishihhsun.com
   ServerAlias www.weishihhsun.com
      
   SSLCertificateFile     /weishihhsun/ssl_key/your certificate.crt
   SSLCertificateKeyFile  /weishihhsun/ssl_key/your certificate key.key
   SSLCertificateChainFile /weishihhsun/ssl_key/your certificate chain.ca-bundle


//After you have enabled SSL, you'll have to restart the web server for the change to be recognized:
sudo service apache2 restart

Monday, September 12, 2016

How to Automatically Scale Up Your Application on AWS ?

This basic rule is you have to make your application servers and databases to be independent of each other. The application servers are only responsible for processing game or business logic without saving any cache data at local. The databases are used for saving game or user data. In front of the application servers, it is necessary to set up a load balancer - while there is any problem occurring to make a server become unavailable - to redirect the user traffic to available servers.  

Moreover, if you want to reduce the overhead of MYSQL such as the number of current connections to improve performance, you can use shared memory servers like Redis or Memcached to save user sessions to decrease the number of SQL queries on MYSQL.

In addition, NoSQL database, Couchbase or MongoDB, would be a better alternative to replace MYSQL database. It can dramatically improve the performance of web applications.

As the user traffic grows, you can easily add one EC2 application server on AWS or  improve MYSQL/NoSQL database specs to meet this need.  

The proposed methods below will teach you how to set up an auto-scale
infrastructure for your applications. 


1.Create StartApp Script for Java Application/Tomcat  at startup time


Initial an EC2 instance and deploy your Java Application on this EC2 and set up a startApp script below at startup time. 

vim startapp

#!/bin/bash

cd /home/weishihhsun/app/
nohup java -jar p2pServer.jar &

sudo cp startapp /etc/init.d

sudo chmod +x /etc/init.d/startapp
sudo update-rc.d startapp defaults 98 02

98 and 02 are the start and stop sequence numbers respectively.
Both are numbers between 00 and 99 and specify how early or late a service is started or killed

By doing so, your Java or Tomcat application would be booted up at startup time.


2. Create an image of EC2 instance on AWS

You need to create an image of EC instance - a clone of this server - by AWS Control UI.    

3. Setup Auto-Scaling Group

You need to add an image of EC2 instance to the auto-scaling group and set up some thresholds to trigger different types of alarms, such as the usage of CPU, hard disk, and internet traffic. 

The usage of CPU, for example, is more than 80%, then automatically increase the EC2 instance of a server; if it is below 10%, then decrease the EC2 instance of  a server.

To conclude, you have learned how to build a high availability and scalability infrastructure for your applications. By making use of the proposed methods, you can automatically scale up your application servers developed by PHP, Java, C# or C/C++ on AWS, as long as you make your applications independent of the database.


Monday, September 5, 2016

How to secure REST API ?

I recommend you implement the protocol of OAuth 2.0 to secure your REST API because the spec of OAuth 2.0 shows that the authorization server must require the use of TLS when sending requests using password authorization as shown below.

HTTP Basic Authorization Request Message:
POST /token HTTP/1.1
     Host: server.example.com
     Content-Type: application/x-www-form-urlencoded

     grant_type=refresh_token&refresh_token=tGzv3JOkF0XG5Qx2TlKWIA
     &client_id=s6BhdRkqt3&client_secret=7Fjfp0ZBr1KtDRbnfVdmIw

Client_id        : Client Account
Client_secret :  Client Password

If you don`t implement TLS/SSL, your plaintext of account and password might be intercepted by Man In Middle Attack.

On the other hand, OAuth 1.0 shows that TLS is must feature to secure your plaintext but did not mention in detail what version of TLS should be adapted or who should implement it

Therefore I propose three methods outlined below to pretend your REST API.

1. Must Use SSL/TLS
You must implement the TLS/SSL before sending the plaintext of password and account to Authorization Server. By doing so, your plaintext would be less vulnerable to being intercepted.

In addition, for telecom application services, they also implement TLS to secure SIP - they set up a secure channel by TLS between the client and the server first and do HTTP Basic Digest Authentication, then start to make, answer or transfer the call in SIP over this secure channel. This is why TLS/SSL is widely used to secure communication layers in the different industry, such as game, telecom, and e-commerce.

2. Shorten the lifetime of a token key
After finishing the authorization, the client will get the access token with the lifetime. You can shorten the lifetime to reduce the risk of a token being hijacked.

3. Implement Your asymmetric encryption algorithm to encrypt plain text values of account and password

What if one day the TLS/SSL becomes not secure, the same situation will happen again that you   must implement your mechanism to encrypt your plaintext of account and password.

Take RSA for example, you can implement the method as shown below.

 Public and private key are generated by your server. You can give the public key to the client in private.

 1. Client :
  tokenUserId = encryption (useId::random key, publicKey)
  tokenPassword = encryption (Password::random key, publicKey)

 The client encrypts the values of userId and Password by public Key and random key and sends  tokenUserId and  tokenPassword to the server.
 The random key could be generated by time stamp or any mechanism you prefer.

  2.Server :
  UserId = decryption (tokenUserId, privateKey)
  Password = decryption (tokenPassword, privateKey)

  The server decrypts those encrypted tokens - tokenUserId and tokenPassword  - with the private key   and starts to do HTTP Basic Authorization.

  Of course, you can use any asymmetric encryption algorithm you like.

 To conclude, the safest method to secure your important data sent over the internet is  using proprietary protocol - Skype, What`s app, Line, for example, all have implemented  their proprietary encryption and decryption algorithms to make the data more secure instead of  following the standard. Perpahs cracking a proprietary protocol is harder than the standard one.  


  Reference :
  RFC 5849 The OAuth 1.0 Protocol
  RFC 6749 The OAuth 2.0 Authorization Framework
  RFC 2617 HTTP Authentication: Basic and Digest Access Authentication
  RFC 5630  The Use of the SIPS URI Scheme in the Session Initiation Protocol (SIP)

Thursday, August 18, 2016

How to secure your Linux ?

Get Started

There are four common and useful methods for you to secure and check your Linux.
  • Port Scan
  • Firewall
  • Update Security package
  • Antivirus

Port Scan

You can check your system by scanning your port number. Once you find out some ports that are open and not used, remember to close them and stop their related services accordingly.
  • Install port scan tool
    sudo install nmap
  • Execute Port Scan command to scan Port from 1 to 65535
    nmap -p 1-65535 -T4 -A -v [Target IP]

Firewall

The fundamental rule to improve the security of your Linux is your system merely opens necessary and required ports for your services. I strongly recommend you to set up restrictions on port 22 to avoid being attacked by SSH Brute Force. Hopefully, DenyHost is a smart firewall that can automatically parse SSH Log and detect malicious IPs to setup block rules for you. 

Update Security package

Update your system with latest security packages automatically.

Install this package if it isn't already installed using
sudo apt-get install unattended-upgrades

To enable it type
sudo dpkg-reconfigure unattended-upgrades

and select "yes".

AntiVirus

Scan and check your system regularly with the following antivirus software.
  • Install Virus
    sudo apt-get install clamav
  • Update Virus code
    sudo freshclam
  • Scan system
    sudo clamscan --remove=yes -i -r ./

Monday, August 15, 2016

Apache Problem due to (24)Too many open files - Solution

After running web services on Apache for few years, my web suddenly could not be accessed because of given an error "HTTP WARNING: HTTP/1.1 403 Forbidden".

When checking the Apache Error log, I found an error message as shown below.

(24)Too many open files: /var/www/html/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable


The solution to this problem is to increase the maximum number of open files and user processes by the following settings.


vim /etc/security/limits.conf 
root soft nofile 32768 
root hard nofile 32768 
root soft nproc 4096  
root hard nproc 4096



Wednesday, August 3, 2016

How to Build an Android Project using Cocos2d-x 3.6 with Eclipse ?

1.Install


Download and install the following libs and SDKs.

    JDK   

        http://www.oracle.com/technetwork/java/javase/downloads/index.html

    Android SDK  

        http://developer.android.com/sdk/installing/index.html?pkg=tools

    Android NDK 10e

        http://dl.google.com/android/repository/android-ndk-r10e-darwin-x86_64.zip

    Apache Ant 

        http://ant.apache.org/bindownload.cgi


2. Configure settings using Eclipse 


Goto Eclipse --> Preference --> C/C++ --> Build --> Environment 

Edit the following settings as shown below.

ANDROID_SDK_ROOT
/Users/User/Applications/Android/sdk

ANR_ROOT
/Users/User/Desktop/cocos2d-x/apache-ant-1.9.7/bin

COCOS_CONSOLE_ROOT
/Users/User/Desktop/cocos2d-x/cocos2d-x-3.6/tools/cocos2d-console/bin

NDK_ROOT
/Users/User/Desktop/cocos2d-x/android-ndk-r10e

PATH
/usr/bin:/bin:/usr/sbin:/sbin:/Users/User/Desktop/cocos2d-x/cocos2d-x-3.6/tools/cocos2d-console/bin





Edit NDK location 





3. Import Cocos2d-x Lib


3.1 In Package Explorer, click import button --> Android --> Existing Android Code Into Workspace 

3.2 Select Cocos2d-x-3.6 --> cocos --> platform --> android --> java





4. Import Source Codes


4.1 Click import button --> Android --> Existing Android Code Into Workspace  --> select your android source codes project 

4.2  Check your project have a link to Cocos2d-x lib

Select your project --> Properties --> Android --> Library 

 



5. Run Configuration 


5.1. Goto Run --> Configurations --> Android Application  --> Tab Android

Name   :  Your Name 
Project :  Click Browse button and select the path of your Source Code

5.2. Goto Run --> Configurations --> Android Application  --> Tab Target 

Run on Android devices 
Check option - Launch on all compatible devices/AVDs

Run on Simulator 
Check option - Automatically pick compatible device : Always uses preferred AVD if set below, launches on compatible device/ AVD otherwise.

5.3. Execute Run button





Tuesday, July 26, 2016

How To Install DenyHosts to defend against Brute Force SSH Attacks on Ubuntu ?

Recently I found there were lots of hackers who used Brute Force SSH Attack Tool to login attempts on my servers. If I check the SSH log file - /var/log/auth.log, I will find too many suspected source IPs to add to a blacklist of firewall.

Hopefully, DenyHosts written in Python is a tool that makes my life easier. It will automatically parse SSH log and setup malicious IPs in a blacklist.

In this tutorial, you will learn how to setup this tool on Ubuntu.


Download and Unzip DenyHosts
wget http://downloads.sourceforge.net/project/denyhost/denyhost-2.8/denyhosts-2.8.tar.gz

Unzip DenyHosts
tar xzf denyhosts*.tar.gz


Install DenyHosts
cd  Denyhosts
sudo python setup.py install

Copy the daemon file to the /etc/init.d/ directory.
sudo cp /usr/local/bin/daemon-control-dist /etc/init.d/denyhosts


Next, open the script and make the below highlighted change
sudo vi /etc/init.d/denyhosts


DENYHOSTS_BIN = “/usr/local/bin/denyhosts.py”

Add white-list IP to the file below
sudo vi /etc/hosts.allow

sshd:Your Server IP

Configuration
sudo vi /etc/denyhosts.conf
# DENY_THRESHOLD_ROOT: block each host after the number of failed
# login attempts has exceeded this value.  This value applies to
# "root" user login attempts only.
#
DENY_THRESHOLD_ROOT = 3



######################################################################
#
# HOSTNAME_LOOKUP
#
# HOSTNAME_LOOKUP=YES|NO
# If set to YES, for each IP address that is reported by Denyhosts,
# the corresponding hostname will be looked up and reported as well
# (if available).
#
HOSTNAME_LOOKUP=YES

# DENY_THRESHOLD_VALID: block each host after the number of failed
# login attempts has exceeded this value.  This value applies to valid
# user login attempts (eg. user accounts that exist in /etc/passwd) except
# for the "root" user
#
DENY_THRESHOLD_VALID = 10

ADMIN_EMAIL = Your Email


Start DenyHosts.
sudo /etc/init.d/denyhosts start

 Stop DenyHosts.
sudo /etc/init.d/denyhosts stop


After setting the DenyHosts and monitoring SSH log for few days, I still found a few of malicious IPs trying to login attempts on my servers. In order to secure my system, it is necessary to change the default port of SSH.

The following information details how to do it.

 
Step 1
As root,  edit the sshd configuration file.
vi /etc/ssh/sshd_config

Step 2
Edit the line which states 'Port 22'.
# What ports, IPs and protocols we listen for
Port 50683


Step 3
Switch over to the new port by restarting SSH.
/etc/init.d/ssh restart

Step 4
Verify SSH is listening on the new port by connecting to it. Note how the port number now needs to be declared.
ssh username@hostname.com -p 50683

The website below allows you to check where your malicious IPs come from.
Abuse IP DB

 

Monday, July 25, 2016

How to setup a TURN Server for your WebRTC application ?

Get Started 
Developing a WebRTC application is easy, but solving the issue of NAT could be a difficult task. If you want to build your WebRTC environment without using any 3rd Party cloud solutions, I think you should take a look at an open source lib -  rfc5766-turn-server. Because it not only supports TURN, but also STUN. 

In this tutorial, you will learn how to set up a STUN & TURN server.

1.Download rfc5766-turn-server package
$ wget http://ftp.cn.debian.org/debian/pool/main/r/rfc5766-turn-server/rfc5766-turn-server_3.2.4.4-1_amd64.deb

2. Install
$ sudo apt-get update
$ sudo apt-get install gdebi-core
$ sudo gdebi rfc5766-turn-server_3.2.4.4-1_amd64.deb

Refer to docs in folder /usr/share/doc/rfc5766-turn-server

vim /opt/etc/turnserver.conf.default


3. Configuration
$ sudo vi /etc/turnserver.conf

// Setup IP address - listening-ip and external-ip are required to set up on EC2 of AWS.
listening-ip=172.31.4.37
external-ip=54.223.149.60

// If TURN server is used for WebRTC,please set long-term credential mechanism as shown below.
lt-cred-mech

// Add a user
user=weishihhsun:mypassword

// Setup realm
realm=mycompany.org

4. Start TURN Server
sudo turnserver -c /usr/local/etc/turnserver.conf --daemo

5. Setup TURN Server`s IP address
"iceServers": [{
   "url": "stun:stun.l.google.com:19302"
}, {
   "url": "turn:54.223.149.60",    // Your Server
   "username": "weishihhsun",  //  your Account
   "credential": "mypassword"   //  Your Password
}]

6. Open firewall`s ports
TCP 443
TCP 3478-3479
TCP 32355-65535
UDP 3478-3479

iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 3478:3479 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 32355:65535 -j ACCEPT
iptables -A INPUT -p udp -m tcp --dport 3478:3479 -j ACCEPT