Scaling Out

For big web systems, scaling is requirement #0.It's part of requirements even it is also architectural concern which is defined as non-functional requirement conventionally.

machine can always be scaled up, but real scalability comes from being able to  scale out.

Key: partition by function and data. asynchronous everywhere.

Best practices summarized from Randy Shoup's good article finding from here: http://www.infoq.com/articles/ebay-scalability-best-practices

partition by function. such as partition by bidding and order tracking.

split horizontally. within same function, break the workload down to manageable units. for example, as logon servers, one server is dedicated for user id between 1 and 10000, second is dedicated for 10001 and 20000 etc.

SOA principle and make services stateless. this makes it easy to scale out since services are equal to each other. it'd be easier to set up load balancer to distribute the work load among them.

Decouple functions asynchronously. A depends on B implies scaling A has to also scaling B, Failure on B implies also failure on A. Methods to use includes queue, batch process such as ETL, multi-cast messaging or other means to relay or staging the request. After decoupling, A can move forward even B is down.

This principle can also be used inside software itself. At every level, decomposing the processing into stages or phases, and connecting them up asynchronously, is critical to scaling.

Anything that can wait should wait. some back end processes should not be part of front end process in order to provide responsive experience to end user. such as transactions in bank system typically be processed in the night. user can quickly submit their requests, but they can endure the latency of waiting couple days for the transactions being synchronized everywhere in the banking system.

Abstraction everywhere. such as defining simple and long lasting interfaces between components. parameters are typically defined as key -value pairs or XML to support extensibility.

"
The virtual machine in many modern languages abstracts the operating system. Object-relational mapping layers abstract the database. Load-balancers and virtual IPs abstract network endpoints. As we scale our infrastructure through partitioning by function and data, an additional level of virtualization of those partitions becomes critical.
"

Using cache appropriately.
not distributed transactions.
databases are put in shards to server different  functions.

Git Notes

https://www.atlassian.com/git/tutorial/git-basics#!init

1. Create a repository
go to the directory you want to treat as central repository and run
git init --bare

e.g.
create repository called amaTest.git in /home/ama/gitRepository:
cd /home/ama/gitRepository
git init --bare amaTest.git

# --bare is to create a shared repository for other person to clone from

to be continued...

Reset Garmin Watches (405, 310xt)

Forerunner 310XT


Soft Reset

1. Connect device to computer (except on Forerunner 310XT)
2. Press and hold Mode and Lap/Reset for 10 seconds
3. Release both buttons
4. Power device on once device starts to charge

Hard Reset (erase user data)

1. Start with device powered off^†
2. Press and hold Mode and Power
3. Release buttons once Do You Really Want to Erase All User Data? message appears
4. Select Yes

Master Hard Reset

1. Power Forerunner off
2. Press and hold Enter and Mode buttons
3. Press and release Power button
4. Press and release Lap/Reset (watch will power off)
5. Wait 3 seconds
6. Release Enter and Mode

Forerunner 405


Soft Reset

1. Connect device to computer using charging clip
2. Press and hold Start/Stop and Lap/Reset for 10 seconds
3. Release and wait 3 seconds for device to power on

Master reset

1. If turned off, power on Forerunner
2. Press and hold Start/Stop and Lap/Reset simultaneously
3. Press both buttons until screen goes blank
4. Release Start/Stop and continue to hold Lap/Reset
5. Release Lap/Reset when the Do you really want erase all user data? message appears
6. Select Enter (Start/Stop) to clear user data

EOF

I keep forget EOF in linux, so even it's simple, it's worth to put here to help me to remember:
unix/linux: ctrl+d
windows:ctrl+z

A Little Bit Design Will Make a Huge Difference

This seems too simple to happen, but it happened in my workplace.

I was reported/complained the database is slow in a load testing. After checking the top queries in database, one statement was identified out easily simply because it was resource heavy it was executed tons of times. The purpose of the statement is to randomly pick a student from a huge student table. It used order by dbms_random.value as a way to randomly pick a record.

The statement is like  this:
--this will sort almost entire table in order to pick one record
select xxx from
(SELECT xxx FROM xxx WHERE xxx ORDER BY dbms_random.value)
where rownum=1;

--rownum=1 is equivelant to rownum<=1. for other values than 1, < and <= are valid operators but not =

In that load testing, it needed to perform testing for 5000 students, and it was designed as a way like for each student, it retrieved a student randomly by executing the statement and then performed corresponding operations on it, thus, 5000 execution of this statement was a result.

One important thing to point out is retrieving student is not part of the testing. But this retrieving represents most of the database load during the testing, this fact itself made the loading testing very inaccurately to reflect the real resource usage of the normal operations.

Without trying to tune the query, I suggested the developer to either prepare a list of student randomly before doing the load testing, or keep a local list of student by invoking that statement just once, and then loop through the list for rest of testing.

The second proposal was adopted because they like to retrieve student as part of the testing, even this would make extra load to the real testing. Retrieving 5000 students is actually the similar cost as to retrieving one student by converting the statement to like where rownum<=5000, but it avoided 4999 times of extra invoking so that the testing would much better reflect the real stress to the systems.

One little  extra thought made the application more reasonable and yet another example of tuning application should happen first before tuning database.

Remote JMX Notification Listener Using Spring Framework

July 22, 2014


We did it easily with pure out-of-box solution in Java. But we want to use Spring since everything else are supported by Spring.

It's simply no working examples on the Internet. Our Senior Java developer said.

I googled and couldn't find meaningful solution either. all the examples are just local listeners that did not help us on our situation.

I eventually decided to invent my own way and fortunately the solution was found within two hours. Thanks for meaningful names used in the framework and my limited but fundamental knowledge on JMX. I guessed I could give it a try when I saw name like NotificationListenerRegistrar. Now lack of example of using Spring to remotely receive JMX notification became a history.

The key is to use org.springframework.jmx.access.NotificationListenerRegistrar. I had no time to verify if its mappedObjectNames is configured properly, but the configuration I used seemed working.

The steps that leads to success is

1. configure a client connector
2. configure a notification listener registrar that picks notification from client connector
3. register my listener in the registrar

Below is the spring context I used in the client side.


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="
    http://www.springframework.org/schema/beans 
    http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/tx 
        http://www.springframework.org/schema/tx/spring-tx.xsd">

<bean id="amaListener" class="ama.test.ConsoleLoggingNotificationListener">
</bean>
<bean id="clientConnector"
class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean">
<property name="serviceUrl"
value="service:jmx:rmi://myIP/jndi/rmi://myIP:1999/jmxrmi" />
</bean>
<--proxy here is not related to notification listener and can be removed --> <bean id="serverProxy" class="org.springframework.jmx.access.MBeanProxyFactoryBean">
<property name="objectName" value="bean:name=amaBean" />
<property name="proxyInterface" value="ama.test.ITestBean" />
<property name="server" ref="clientConnector" />
</bean>
<bean id="notificationListener" class="org.springframework.jmx.access.NotificationListenerRegistrar">
<property name="mappedObjectNames">
<list>
<value>bean:name=amaBean</value>
</list>
</property>
<property name="NotificationListener" ref="amaListener" />
<property name="server" ref="clientConnector" />
</bean>


</beans>



Below are the whole examples including both MBean and client side listener. The source code of MBean was copied from web example so that it must be familiar to you if you already searched on this topic. BTW, the Spring I used is version 4. And one last piece, the VM parameters passed in while starting the MBean application (SpringMain.java, not for TestClient.java) are like this:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=myIP
BTW, if port is specified, for instance, -Dcom.sun.management.jmxremote.port=2014, then you defined a standard way to access JMX server, and you can connect to JMX server via both 1999 and 2014 in this example.

ITestBean.java The interface

package ama.test;

public interface ITestBean {
public int add(int x, int y);
}

JmxTestBean.java

package ama.test;

import org.springframework.jmx.export.notification.NotificationPublisherAware;
import org.springframework.jmx.export.notification.NotificationPublisher;
import javax.management.Notification;

public class JmxTestBean implements NotificationPublisherAware, ITestBean {

    private String name;
    public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

private int age;
    private boolean isSuperman;
    private NotificationPublisher publisher;

//Its add method can be invoked via jconsole and it sends out notification.

    public int add(int x, int y) {
        int answer = x + y;
        this.publisher.sendNotification(new Notification("add", this, 0));
        return answer;
    }

    public void dontExposeMe() {
        throw new RuntimeException();
    }
    
    public void setNotificationPublisher(NotificationPublisher notificationPublisher) {
        this.publisher = notificationPublisher;
    }
}

SpringMain.java
This is the application that has an M Bean defined.

package ama.test;

import org.springframework.context.support.ClassPathXmlApplicationContext;

public class SpringMain {

  public static void main(String[] args) throws Exception{

       new ClassPathXmlApplicationContext("application-context.xml");
       System.in.read();
    }
}

TestClient.java
--This client is used to demonstrate remove notification listener by using Spring framework.
package ama.test;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class TestClient {

 public static void main(String[] args) throws Exception{
 
      new ClassPathXmlApplicationContext("client-app-context.xml");
       System.in.read();
   }

}

application-context.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:tx="http://www.springframework.org/schema/tx"
    xsi:schemaLocation="
    http://www.springframework.org/schema/beans 
    http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/tx 
        http://www.springframework.org/schema/tx/spring-tx.xsd">
    
   
<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean">
<property name="locateExistingServerIfPossible" value="true" />
</bean>
<bean id="exporter" class="org.springframework.jmx.export.MBeanExporter">
<property name="beans">
<map>
<entry key="bean:name=amaBean" value-ref="testBean" />
</map>
</property>
<property name="server" ref="mbeanServer"/>
<!--
<property name="notificationListenerMappings">
<map>
<entry key="bean:name=amaBean">
<bean class="ama.test.ConsoleLoggingNotificationListener" />
</entry>
</map>
</property>
-->
</bean>

  <bean id="testBean" class="ama.test.JmxTestBean">
    <property name="name" value="TEST"/>
    <property name="age" value="100"/>
  </bean>

<bean id="registry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
      <property name="port" value="1999"/>
   </bean>

   <bean id="serverConnector"
         class="org.springframework.jmx.support.ConnectorServerFactoryBean"
         depends-on="registry">
      <property name="objectName" value="connector:name=rmi"/>
      <property name="serviceUrl" 
                value="service:jmx:rmi://myIP/jndi/rmi://myIP:1999/jmxrmi"/>
   </bean>
</beans>

client-app-context.xml

  <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd"> <bean id="amaListener" class="ama.test.ConsoleLoggingNotificationListener"> </bean> <bean id="clientConnector" class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean"> <property name="serviceUrl" value="service:jmx:rmi://10.2.39.108/jndi/rmi://10.2.39.108:1999/jmxrmi" /> </bean> <bean id="serverProxy" class="org.springframework.jmx.access.MBeanProxyFactoryBean"> <property name="objectName" value="bean:name=amaBean" /> <property name="proxyInterface" value="ama.test.ITestBean" /> <property name="server" ref="clientConnector" /> </bean> <bean id="notificationListener" class="org.springframework.jmx.access.NotificationListenerRegistrar"> <property name="mappedObjectNames"> <list> <value>bean:name=amaBean</value> </list> </property> <property name="NotificationListener" ref="amaListener" /> <property name="server" ref="clientConnector" /> </bean> </beans>

This along with this post finished my day perfectly. :)

3 Pillars of Information Security

Business Security Services
--driven by regulation concerns, partnerships etc
IT Security Services
--responsible for addressing how the business services are physically deployed.
Security Policy Management
--ensures business security services are managed in a consistent manner with IT. It links the business-related andIT-related security servies together.

From The Art of Enterprise Information Architecture.