MongoDB keyfile permissions are too open error

Creating an authenticated replicated set requires setting up of a secret key file on all the nodes.
In spite of creating the key file and putting the same file on all the nodes, you may encounter error stating that the permissions are too open:

permissions on /var/mongodb/conf/mongodb-keyfile are too open

 To solve this, here are the two things that you need to do to solve it:

1.  Check that the owner of the key file and the user running mongod are same.
    Use chown command to change ownership to match if required.
     chown
2.  Check permissions of the key file - should be 600 or 700.
    Use chmod command to make permissions more restrictive.
    chmod 600

Amazon Echo - your own personal digital concierge!

Amazon has launched nifty device named Echo - which is like a stand-alone "Siri" - but more like a personal digital concierge for the family. The cost is pretty reasonable at $199 and Amazon is running a promotion right now where Prime members can get it for only $99.

Check out the Echo on Amazon - looks pretty interesting but is available by invite only.

MongoDB warning for running on NUMA machine

When running MongoDB on a Linux server (having multiple physical processors as most servers have these days), you will see the below warning in your log file on startup:

2014-09-04T13:13:35.245-0400 [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-09-04T13:13:35.245-0400 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-09-04T13:13:35.245-0400 [initandlisten] **              numactl --interleave=all mongod [other options]

This check for NUMA based system is done on startup since MongoDB2.0 as running MongoDB or for that matter any database applications can cause poor performance due to excessive swapping.

There is no ideal solution to this problem yet, but a good solution would be to switch off zone reclaim and set a memory interleave policy before starting mongod process. Here are the two steps, both of which need to be done to fully disable NUMA:

1. Switch off zone reclaim:
   echo 0 > /proc/sys/vm/zone_reclaim_mode
2. Flush Linux buffer caches just before starting mongod:
   sysctl -q -w vm.drop_caches=3
3. Set interleave policy for mongod process when starting it:
   numactl --interleave=all <$MONGO_HOME>/mongod <options>

This should solve your problem - but if you want to understand what the issue is and what's really going on - then continue to read below!

Older multi-processor systems were designed to have Uniform Memory Access (UMA) - this essentially means is that all memory is common to all processors and accessed via a common bus. Thus the latency and performance of all processors was the same. This is also referred to as Symmetric Multi-Processing (SMP).

New multi-processor systems were designed to have Non-Uniform Memory Access (NUMA) - each processor has a local bank of memory to which it has very fast access & low latency and it also has access to memory banks of other processors though with a poorer performance & higher latency. This is sometimes also referred to as Cache-Coherent NUMA (ccNUMA).

Linux is aware of its running on a NUMA system and does the following:

• Probes the system to figure out the physical layout
• Attaches the memory module to its' corresponding processor to create a node
• Creates a map (table/matrix) of the cost or weight of inter-node communication between all the nodes
• Allocates a preferred node for each thread to run on and allocates memory to it, preferably from the same node or from another node if available. If memory is not available on the same node, it is allocated from the available node with the lowest cost
• Uses zone reclaim to reclaim memory when a node runs out of memory

Zone Reclaim allows Linux SysAdmins to set a more or less aggressive approach to reclaim memory when a zone runs out of memory. If it is set to zero, which is the default, then no zone reclaim occurs. Allocations will be satisfied from other zones / nodes in the system.

Now that you have a simplistic view of what NUMA is, let's understand what our solution actually did.

1. We switched off zone reclaim - this would reduce paging as allocation of memory is now satisfied from other nodes if required
2. Flushed Linux's buffer caches - This helps to ensure allocation fairness, even if the daemon is restarted while significant amounts of data are in the operating system buffer cache.
3. We set memory allocation to interleave for mongod - this spreads allocation of memory evenly on a round-robin basis across all nodes thus spreading out memory usage and reducing paging

For a more in-depth analysis and further reading, check out Jeremy Cole's blog posts:

• The MySQL “swap insanity” problem and the effects of the NUMA architecture
• A brief update on NUMA and MySQL
• Zone Reclaim Mode (search for zone_reclaim on this page)

Maven Compile error

Maven Compile Plugin by default uses version 2.0.2 and JDK1.3 as the target for the compilation of Java code if you do not specify the compiler plugin version and the Java source & target versions. Due to this, you may get the below error when compiling:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project scratch: Compilation failure
[ERROR] /home/projects/java/scratch/src/main/java/com/test/MavenCompileTest.java:[50,26] error: for-each loops are not supported in -source 1.3

To resolve this problem, you need to specify the compiler plugin version and the Java -source and -target versions in the pom.xml of your project as below:

  
    
      org.apache.maven.plugins
      maven-compiler-plugin
      3.1
      
        1.7
        1.7
      
    
  

You can figure out the latest version of the Compiler Plugin by reading the version number on the top right of the Compiler Plugin page.

Throttle task submission in Java - Simple solution

JDK provides a convenient java.util.concurrent.Executors factory with useful methods to return various java.util.concurrent.Executor implementation instance pre-configured with commonly used settings. However, there is no implementation or configuration available for an implementation that throttles task submission. Here I provide my take on a very simple solution for this requirement.

private void submitTasks(List tasksList) {
  final ThreadPoolExecutor executor = new ThreadPoolExecutor(MIN_POOL_SIZE,
                                                             MAX_POOL_SIZE,
                                                             60L,
                                                             TimeUnit.SECONDS,
                                                             new LinkedBlockingQueue());
  for (Runnable task : tasksList) {
    while (executor.getQueue().size() > MAX_Q_SIZE) {
      // Throttle for WAIT_FOR_SECONDS seconds if queue is full
      try {
        log.info("Waiting " + WAIT_FOR_SECONDS + " seconds as Queue size is: " + executor.getQueue().size());
        Thread.sleep(WAIT_FOR_SECONDS * 1000);
      } catch (InterruptedException e) {
        // Ignore
      }
    }
    executor.execute(task);
  }
  // inform the executor there are no more tasks
  executor.shutdown();
}

Instead of using the factory to get a pre-configured executor, I create my own ThreadPoolExecutor using a LinkedBlockingQueue. This allows us access to the underlying queue for the throttling feature. Before submitting the task, I check if the current queue size is greater than a certain size and if so, then wait for a configured time repeatedly till the queue is smaller.
This implementation is very useful for the scenarios where you have lots of short running tasks that need to be processed by a fixed small number of threads.

The mystery of the missing WhiteSpace in XML Attribute values

My expectation when testing the component that handles the import & export of xml data for the system, was that the input & output xml data should be the same if the component works properly. However, I noticed minor whitespace differences in attribute values between the two xml files and I couldn't find any code that was trimming attribute values during either the import nor the export of xml. A closer look showed that the spaces in attribute values were being trimmed and multiple occurrences of space replaced by a single space when the attribute values were being read in!

This was very intriguing so digging around in the XML Specs, says that in the canonical form of an XML document, attribute values are normalized by the XML processor. The Attribute Value Normalization section further lists out the exact algorithm to be used by the XML processor where-in all occurrences of whitespace are replaced by a space. Furthermore, if the attribute type is not CDATA, then the XML processor must further process the normalized attribute value by trimming leading and trailing space and by replacing sequences of spaces by a single space. Also, there is a separate section to handle new line characters which states that all line-breaks or occurrences of CR & LF must be replaced by a LF character.