Moved to Game Tech Tutorial

 Hi 

In recent years, I created a website called Game Tech Tutral to share experiences and solutions in AWS, GCP, Azure, DevOps, SDK, Mobile, and Website development. 

If you are interested in those technologies, welcome to visit us at our website: Game Tech Tutorial below or follow us on Facebook to get the latest tutorials.

Thank you. 

How to extract Unity package?

Extract with Python3 

pip3 install unitypackage_extractor

python3 -m unitypackage_extractor [path/to/your/package.unitypackage]

ref:

https://github.com/Cobertos/unitypackage_extractor

X509: Certificate Signed by Unknown Authority & Go Docker & EKS

 If you encountered an issue below, your go application on EKS failed to send an HTTP request to other services.

Problem:x509: certificate signed by unknown authority

This is due to fact that your HTTP library failed to read the CA certificate in setting up SSL communication with other services. Then we can suspect missing or incorrect CA certificate is the cause of this problem.

Debug Step:

• Check your ca-certificates are packed to the Docker image or not.
• If not, you can install ca-certificates as below in the DockerFile.

RUN apk add --update --no-cache ca-certificates

If you use multiple-stage to build go application to reduce the size of the docker image, remember to add the whole folder /etc/ssl/certs to your docker image as below.

FROM scratch

WORKDIR /COPY --from=builder /etc/ssl/certs./etc/ssl/certs

How to build a lightweight go application with CA certificates in DockerFile for EKS?

ref:

https://tyricwei.medium.com/x509-certificate-signed-by-unknown-authority-go-docker-eks-f508a49d86f6

Solutions to Solve Writing Contention Problems in Google Datastore

Google Cloud Platform provides lots of technologies to save developer`s time while thinking about how to scale or maintain our services to serve millions of thousands of users. If you are familiar with MYSQL`s architecture, you will face different kinds of problems to serve a high volume of users. The reason is MYSQL implements the blocking mode when writing data to databases. Google Datastore solved this issue and can write a lot of data in parallel. Especially, Google Datastore is designed for high availability and scalability. 

However, you must test and know deeply about how to write simultaneous data in datastore. Otherwise, your services could only serve current 5 requests per second. It seems like a buggy solution. What is worst, Google's documentations do not give you an easy-to-understand example to implement Sharding Counter. You may wonder how to apply this technique to your entity model and how to avoid this common mistake.

Limitations of Google Datastore

• Any entity group can only be written at rate 1 request per second.
• If using @ndb.transactional or @ndb.transactional(xg=true) to write the data, your API can only serve current 5 requests per second. Otherwise, you will get an error of writing contention in datastore.

Why is writing data in Datastore so slow?

Because Datastore needs to copy your data globally and make your data with high availability.

Solutions to Solve Writing Contention Problems

• Sharding Counter 

• Use Memcache to batch writing requests and do all the operations in memory and return back to your clients

• Defer a task queue to write data in datastore

In fact, Sharding Counter is just an example provided by Google. The key point is we can do sharding on our entity group with a unique id as shown below. 

The following codes show how to simultaneously write a thousand of Friendship entities in parallel. If we need to improve its performance, just increase the number of NUM_SHARDS.

NUM_SHARDS = 1000
shard_string_index = str(random.randint(0, NUM_SHARDS - 1))

FriendShip(id=shard_string_index,

           user_key='user Id', 

           friend_key='frind Id')

If you have lots of data models needed to update in one request, please use the task queue to update and return back to your clients only a few amount of information.

If you need to write transactional data using @ndb.transactional or @ndb.transactional(xg=true), defer a task queue to get it done and return a few amount of information to the clients.

Depending on your data model's design, you can use Sharding Counter or Memcache or Task Queue or a hybrid approach to performing the best performance using Google's Datastore and Google App Engine.

How to make TURN Server for high availability?

If you want to keep your WebRTC video streaming services online without any downtime, you must pay attention to the availability of TURN Server. Because TURN Server plays an important to help two parties to connect to each other with Video or Audio streaming in different NAT networks.

The following instructions show how to automatically monitor your TURM server and restart it during the downtime.


1. Install pexpect lib in Python 

sudo pip install pexpect --upgrade



2. Edit MonitorStun.py 
- Telnet your TURN Serer 
- If it is down, ssh to your server and restart it  

#!/usr/bin/env python

import socket

import subprocess

import sys

from datetime import datetime

from pexpect import pxssh

# SSH TO TURN SERVER and restart it

def connect_turn_server():

  s = pxssh.pxssh()

  if not s.login ('TURN Server IP', 'SERVER PORT', 'ACCOUNT', 'PASSWORD'):

    print "SSH session failed on login."

    print str(s)

  else:

    print "SSH session login TURN successful"

    s.sendline ('sudo turnserver -c /usr/local/etc/turnserver.conf --daemo')

    s.prompt()         # match the prompt

    print s.before     # print everything before the prompt.

    s.logout()

# Telnet TURN Server to check it is alive or not on PORT 3478 or 3479

# Clear the screen

subprocess.call('clear', shell=True)

# Ask for input

remoteServer    = 'SERVER IP'

remoteServerIP  = socket.gethostbyname(remoteServer)

# Print a nice banner with information on which host we are about to scan

print "-" * 60

print "Please wait, scanning remote host", remoteServerIP

print "-" * 60

# Check what time the scan started

t1 = datetime.now()

# Using the range function to specify ports (here it will scans all ports between 1 and 1024)

# We also put in some error handling for catching errors

try:

    for port in range(3478,3479):

        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

        result = sock.connect_ex((remoteServerIP, port))

        if result == 0:

            print "Port {}:      Open".format(port)

        else:

             print "TURN Server is down"

             connect_turn_server()

             print "restart TURN Server OK"

        sock.close()

except KeyboardInterrupt:

 print "You pressed Ctrl+C"

    sys.exit()

except socket.error:

    print "Couldn't connect to server"

    sys.exit()

                                        

3. Add MonitorStun.py to con job to check TURN Server in every 1 min.

*/1 * * * * /your_path/monitorStun.py

Of course, you can apply this technique to monitor any services such as SIP Proxy with port 5060, Apache with port 80, or Tomcat with port 8080.

How to setup a load balancing and failover back-end RDS for WordPress ?

WordPress is widely used to develop webs for multiple platforms. 
It is necessary to set up WordPress to deploy on a load balancing and failover backend RDS if you want to provide a service without any downtime for your customers.

How to get this  done ?

1. Create a Master and Slave database on RDS.

2.Import HyperDB Plugin into WordPress folder.

HyperDB can be download at below.
https://wordpress.org/plugins/hyperdb/


3. Setup the Master and Salve URL at HyperDB`s config file.


4. Performance Test 

The following info shows the test result on AWS.

EC2         Current session                RDS                                           

t2.micro            110 / s                     single master        

t2.medium         210 /s                       one master and one slave    


To conclude, using WordPress framework to develop the website can achieve a high performance at 210 current requests per second on EC2 with t2.medium spec; with t2.micro of EC2, the best throughput can also achieve 110 current requests per second. Thus, I think WordPress is good enough to build a high-performance website by PHP on AWS.

Setting up Apache Server with SSL

//Enable the module by typing:

sudo a2enmod ssl

//  Configure SSL
sudo vim /etc/apache2/sites-available/default-ssl.conf

   ServerAdmin your gmail
   ServerName weishihhsun.com
   ServerAlias www.weishihhsun.com
      
   SSLCertificateFile     /weishihhsun/ssl_key/your certificate.crt
   SSLCertificateKeyFile  /weishihhsun/ssl_key/your certificate key.key
   SSLCertificateChainFile /weishihhsun/ssl_key/your certificate chain.ca-bundle


//After you have enabled SSL, you'll have to restart the web server for the change to be recognized:
sudo service apache2 restart

How to Automatically Scale Up Your Application on AWS ?

This basic rule is you have to make your application servers and databases to be independent of each other. The application servers are only responsible for processing game or business logic without saving any cache data at local. The databases are used for saving game or user data. In front of the application servers, it is necessary to set up a load balancer - while there is any problem occurring to make a server become unavailable - to redirect the user traffic to available servers.  

Moreover, if you want to reduce the overhead of MYSQL such as the number of current connections to improve performance, you can use shared memory servers like Redis or Memcached to save user sessions to decrease the number of SQL queries on MYSQL.

In addition, NoSQL database, Couchbase or MongoDB, would be a better alternative to replace MYSQL database. It can dramatically improve the performance of web applications.

As the user traffic grows, you can easily add one EC2 application server on AWS or  improve MYSQL/NoSQL database specs to meet this need.  

The proposed methods below will teach you how to set up an auto-scale
infrastructure for your applications. 

1.Create StartApp Script for Java Application/Tomcat  at startup time

Initial an EC2 instance and deploy your Java Application on this EC2 and set up a startApp script below at startup time. 
vim startapp

#!/bin/bash
cd /home/weishihhsun/app/
nohup java -jar p2pServer.jar &

sudo cp startapp /etc/init.d
sudo chmod +x /etc/init.d/startapp
sudo update-rc.d startapp defaults 98 02

98 and 02 are the start and stop sequence numbers respectively.
Both are numbers between 00 and 99 and specify how early or late a service is started or killed

By doing so, your Java or Tomcat application would be booted up at startup time.

2. Create an image of EC2 instance on AWS

You need to create an image of EC instance - a clone of this server - by AWS Control UI.    

3. Setup Auto-Scaling Group

You need to add an image of EC2 instance to the auto-scaling group and set up some thresholds to trigger different types of alarms, such as the usage of CPU, hard disk, and internet traffic. 

The usage of CPU, for example, is more than 80%, then automatically increase the EC2 instance of a server; if it is below 10%, then decrease the EC2 instance of  a server.

To conclude, you have learned how to build a high availability and scalability infrastructure for your applications. By making use of the proposed methods, you can automatically scale up your application servers developed by PHP, Java, C# or C/C++ on AWS, as long as you make your applications independent of the database.

How to secure REST API ?

I recommend you implement the protocol of OAuth 2.0 to secure your REST API because the spec of OAuth 2.0 shows that the authorization server must require the use of TLS when sending requests using password authorization as shown below.

HTTP Basic Authorization Request Message:
POST /token HTTP/1.1
     Host: server.example.com
     Content-Type: application/x-www-form-urlencoded

     grant_type=refresh_token&refresh_token=tGzv3JOkF0XG5Qx2TlKWIA
     &client_id=s6BhdRkqt3&client_secret=7Fjfp0ZBr1KtDRbnfVdmIw

Client_id        : Client Account
Client_secret :  Client Password

If you don`t implement TLS/SSL, your plaintext of account and password might be intercepted by Man In Middle Attack.

On the other hand, OAuth 1.0 shows that TLS is must feature to secure your plaintext but did not mention in detail what version of TLS should be adapted or who should implement it

Therefore I propose three methods outlined below to pretend your REST API.

1. Must Use SSL/TLS
You must implement the TLS/SSL before sending the plaintext of password and account to Authorization Server. By doing so, your plaintext would be less vulnerable to being intercepted.

In addition, for telecom application services, they also implement TLS to secure SIP - they set up a secure channel by TLS between the client and the server first and do HTTP Basic Digest Authentication, then start to make, answer or transfer the call in SIP over this secure channel. This is why TLS/SSL is widely used to secure communication layers in the different industry, such as game, telecom, and e-commerce.

2. Shorten the lifetime of a token key
After finishing the authorization, the client will get the access token with the lifetime. You can shorten the lifetime to reduce the risk of a token being hijacked.

3. Implement Your asymmetric encryption algorithm to encrypt plain text values of account and password

What if one day the TLS/SSL becomes not secure, the same situation will happen again that you   must implement your mechanism to encrypt your plaintext of account and password.

Take RSA for example, you can implement the method as shown below.

 Public and private key are generated by your server. You can give the public key to the client in private.

 1. Client :
  tokenUserId = encryption (useId::random key, publicKey)
  tokenPassword = encryption (Password::random key, publicKey)

 The client encrypts the values of userId and Password by public Key and random key and sends  tokenUserId and  tokenPassword to the server.
 The random key could be generated by time stamp or any mechanism you prefer.

  2.Server :
  UserId = decryption (tokenUserId, privateKey)
  Password = decryption (tokenPassword, privateKey)

  The server decrypts those encrypted tokens - tokenUserId and tokenPassword  - with the private key   and starts to do HTTP Basic Authorization.

  Of course, you can use any asymmetric encryption algorithm you like.

 To conclude, the safest method to secure your important data sent over the internet is  using proprietary protocol - Skype, What`s app, Line, for example, all have implemented  their proprietary encryption and decryption algorithms to make the data more secure instead of  following the standard. Perpahs cracking a proprietary protocol is harder than the standard one.  

  Reference :
  RFC 5849 The OAuth 1.0 Protocol
  RFC 6749 The OAuth 2.0 Authorization Framework
  RFC 2617 HTTP Authentication: Basic and Digest Access Authentication
  RFC 5630  The Use of the SIPS URI Scheme in the Session Initiation Protocol (SIP)

How to secure your Linux ?

Get Started

There are four common and useful methods for you to secure and check your Linux.

• Port Scan
• Firewall
• Update Security package
• Antivirus

Port Scan

You can check your system by scanning your port number. Once you find out some ports that are open and not used, remember to close them and stop their related services accordingly.

• Install port scan tool
  sudo install nmap
  
• Execute Port Scan command to scan Port from 1 to 65535
  nmap -p 1-65535 -T4 -A -v [Target IP]
  

Firewall

The fundamental rule to improve the security of your Linux is your system merely opens necessary and required ports for your services. I strongly recommend you to set up restrictions on port 22 to avoid being attacked by SSH Brute Force. Hopefully, DenyHost is a smart firewall that can automatically parse SSH Log and detect malicious IPs to setup block rules for you. 

• Install DenyHost

Update Security package

Update your system with latest security packages automatically.

Install this package if it isn't already installed using
sudo apt-get install unattended-upgrades

To enable it type
sudo dpkg-reconfigure unattended-upgrades

and select "yes".

AntiVirus

Scan and check your system regularly with the following antivirus software.

• Install Virus
  sudo apt-get install clamav
  
• Update Virus code
  sudo freshclam
  
• Scan system
  sudo clamscan --remove=yes -i -r ./
  

Apache Problem due to (24)Too many open files - Solution

After running web services on Apache for few years, my web suddenly could not be accessed because of given an error "HTTP WARNING: HTTP/1.1 403 Forbidden".

When checking the Apache Error log, I found an error message as shown below.

(24)Too many open files: /var/www/html/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

The solution to this problem is to increase the maximum number of open files and user processes by the following settings.


vim /etc/security/limits.conf 
root soft nofile 32768 
root hard nofile 32768 
root soft nproc 4096  
root hard nproc 4096

How to Build an Android Project using Cocos2d-x 3.6 with Eclipse ?

1.Install

Download and install the following libs and SDKs.

    JDK   

        http://www.oracle.com/technetwork/java/javase/downloads/index.html

    Android SDK  

        http://developer.android.com/sdk/installing/index.html?pkg=tools

    Android NDK 10e

        http://dl.google.com/android/repository/android-ndk-r10e-darwin-x86_64.zip

    Apache Ant 

        http://ant.apache.org/bindownload.cgi

2. Configure settings using Eclipse 

Goto Eclipse --> Preference --> C/C++ --> Build --> Environment 

Edit the following settings as shown below.

ANDROID_SDK_ROOT

/Users/User/Applications/Android/sdk

ANR_ROOT

/Users/User/Desktop/cocos2d-x/apache-ant-1.9.7/bin

COCOS_CONSOLE_ROOT

/Users/User/Desktop/cocos2d-x/cocos2d-x-3.6/tools/cocos2d-console/bin

NDK_ROOT

/Users/User/Desktop/cocos2d-x/android-ndk-r10e

PATH

/usr/bin:/bin:/usr/sbin:/sbin:/Users/User/Desktop/cocos2d-x/cocos2d-x-3.6/tools/cocos2d-console/bin

Edit NDK location 

3. Import Cocos2d-x Lib

3.1 In Package Explorer, click import button --> Android --> Existing Android Code Into Workspace 

3.2 Select Cocos2d-x-3.6 --> cocos --> platform --> android --> java

4. Import Source Codes

4.1 Click import button --> Android --> Existing Android Code Into Workspace  --> select your android source codes project 

4.2  Check your project have a link to Cocos2d-x lib

Select your project --> Properties --> Android --> Library 

 

5. Run Configuration 

5.1. Goto Run --> Configurations --> Android Application  --> Tab Android

Name   :  Your Name 

Project :  Click Browse button and select the path of your Source Code

5.2. Goto Run --> Configurations --> Android Application  --> Tab Target 

Run on Android devices 

Check option - Launch on all compatible devices/AVDs

Run on Simulator 

Check option - Automatically pick compatible device : Always uses preferred AVD if set below, launches on compatible device/ AVD otherwise.

5.3. Execute Run button

How to setup a TURN Server for your WebRTC application ?

Get Started 

Developing a WebRTC application is easy, but solving the issue of NAT could be a difficult task. If you want to build your WebRTC environment without using any 3rd Party cloud solutions, I think you should take a look at an open source lib -  rfc5766-turn-server. Because it not only supports TURN, but also STUN. 

In this tutorial, you will learn how to set up a STUN & TURN server.

1.Download rfc5766-turn-server package

$ wget http://ftp.cn.debian.org/debian/pool/main/r/rfc5766-turn-server/rfc5766-turn-server_3.2.4.4-1_amd64.deb

2. Install

$ sudo apt-get update

$ sudo apt-get install gdebi-core

$ sudo gdebi rfc5766-turn-server_3.2.4.4-1_amd64.deb

Refer to docs in folder /usr/share/doc/rfc5766-turn-server

vim /opt/etc/turnserver.conf.default

3. Configuration

$ sudo vi /etc/turnserver.conf

// Setup IP address - listening-ip and external-ip are required to set up on EC2 of AWS.

listening-ip=172.31.4.37

external-ip=54.223.149.60

// If TURN server is used for WebRTC，please set long-term credential mechanism as shown below.

lt-cred-mech

// Add a user

user=weishihhsun:mypassword

// Setup realm

realm=mycompany.org

4. Start TURN Server

sudo turnserver -c /usr/local/etc/turnserver.conf --daemo

5. Setup TURN Server`s IP address

"iceServers": [{

   "url": "stun:stun.l.google.com:19302"

}, {

   "url": "turn:54.223.149.60",    // Your Server

   "username": "weishihhsun",  //  your Account

   "credential": "mypassword"   //  Your Password

}]

6. Open firewall`s ports

TCP 443

TCP 3478-3479

TCP 32355-65535

UDP 3478-3479

iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

iptables -A INPUT -p tcp -m tcp --dport 3478:3479 -j ACCEPT

iptables -A INPUT -p tcp -m tcp --dport 32355:65535 -j ACCEPT

iptables -A INPUT -p udp -m tcp --dport 3478:3479 -j ACCEPT

How to solve a network problem ?

Have you ever had a network problem due to packet lost or some unknown issues in your routers ? How could you find out those issues may be caused by your routers or Internet Service Provider ?

The following instructions will guide you how to figure it out.

// 1. Download mtr

http://rudix.org/packages/mtr.html

// 2. Execute the following command to check whether or not there is a packet lost problem in your network node.

sudo mtr -c 100 -r [target IP]

// 3. Then execute traceroute

traceroute [target IP]

You will know that the network node is shown in IP address or Domain Name.

If it is IP address, it probably could be the router; if it is the domain name, it might be your ISP`s network devices.

// 4 Download nmap
https://nmap.org/download.html

// 5. Dump router`s info and you can find out what your router is, so as to fix it.

nmap -A -T4 router IP

Ex:

EricWeis-MacBook-Air:~ EricWei$ nmap -A -T4 192.168.11.1

Starting Nmap 7.12 ( https://nmap.org ) at 2016-07-07 23:33 JST

Stats: 0:01:47 elapsed; 0 hosts completed (1 up), 1 undergoing Service Scan

Service scan Timing: About 80.00% done; ETC: 23:35 (0:00:27 remaining)

Nmap scan report for buffalo.setup (192.168.11.1)

Host is up (0.027s latency).

Not shown: 995 closed ports

PORT      STATE SERVICE VERSION

53/tcp    open  domain  (status request not implemented)

80/tcp    open  http    Buffalo AirStation http config

| http-auth:

| HTTP/1.0 401 Unauthorized

|_  Basic realm=AirStation

|_http-title: UNAUTHORIZED

2601/tcp  open  zebra   Quagga routing software

2602/tcp  open  zebra   Quagga routing software

49152/tcp open  upnp    Intel UPnP reference SDK 1.2 (Linux 2.4.20; UPnP 1.0)

1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :

SF-Port53-TCP:V=7.12%I=7%D=7/7%Time=577E6838%P=x86_64-apple-darwin13.4.0%r

SF:(DNSVersionBindReq,20,"\0\x1e\x02G\x81\x85\0\x01\0\0\0\0\0\0\x07version

SF:\x04bind\0\0\x10\0\x03")%r(DNSStatusRequest,E,"\0\x0c\x02H\x90\x84\0\0\

SF:0\0\0\0\0\0");

Service Info: OS: Linux; Device: WAP; CPE: cpe:/o:linux:linux_kernel:2.4.20

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 141.69 seconds