If I only had a ruler, I would rule the world.

What would you need to have a system that never fails

This entry is still a work in progress - but publishing half finished as I need the links.

• https://github.com/Netflix/servo/
• http://metrics.codahale.com/
• https://github.com/etsy/statsd/
• https://github.com/youdevise/java-statsd-client

Graphing:

• http://graphite.wikidot.com/
• http://stackoverflow.com/questions/10527401/graphite-is-ugly-is-there-a-prettier-interface-for-statsd-data
• http://square.github.io/cubism/
•  

... Java so fast it blows women's cloths off!

The question I am trying to answer is what does it take to get Java to perform so fast it blows women's cloths off. The goal is not to understand what is fast - but simply to look at techniques which are faster than anything else available.

1. Never use java.util (collections). They generate tremendous amount of garbage and are slow.

• Trove
• Colt
• If you don't believe me - http://b010.blogspot.nl/2009/05/speed-comparison-of-1-javas-built-in.html   

2. Avoid garbage collection

• http://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html - No minor GC collections until Eden fills up - so make your Eden larger. (-XX:MaxNewSize)
• http://java-performance.info/overview-of-memory-saving-techniques-java/

 3. Reuse strings.

• http://java-performance.info/string-intern-in-java-6-7-8/
• https://github.com/OpenHFT/Java-Lang/blob/master/lang/src/main/java/net/openhft/lang/pool/StringInterner.java - very quick way to do on heap intern

3. Async Logging - i.e. don't spend all your time writing logs

• http://logging.apache.org/log4j/2.x/manual/async.html
• https://github.com/OpenHFT/Java-Chronicle - record everything for high-frequency-trading (read fast enough to blow women's cloths off)

4. Don't use the Heap - i.e. No heap = no GC.

• https://github.com/OpenHFT/HugeCollections - off heap storage
• https://github.com/OpenHFT/Java-Lang

5.  Know what the heck your system is doing. Maybe it's the machine and not your crappy-code?

• http://www.azulsystems.com/jhiccup - unexplained processing pauses
• http://dtrace.org/blogs/brendan/2013/06/08/linux-performance-analysis-and-tools/ - tools to monitor Linux
• http://www.joyent.com/blog/the-new-systems-performance - some more tools

6. Use a ring instead of a queue / thread message passing

• http://lmax-exchange.github.io/disruptor/ - inter thread message passing.

7. Maybe the problem isn't with your code at all - but with the watch you use?

• https://github.com/hajile/microclock - library implementing RDTSC (let's count the CPU clock cycles)
• http://www.strchr.com/performance_measurements_with_rdtsc
• https://github.com/giltene/HdrHistogram - more precise measurements

8. Keep your methods short and sweet - helps with hotspot

•  Write code that is easy for the optimizer - https://wikis.oracle.com/display/HotSpotInternals/PerformanceTechniques

9. Try some exotic features

• https://github.com/OpenHFT/Java-Thread-Affinity - Thread affinity
• http://www.docjar.com/docs/api/sun/misc/Unsafe.html - a magical class in the JDK, which does everything a magnitude faster, but don't use it without protection. It's unsafe. 
• http://architects.dzone.com/articles/getting-cc-performance-java - serialization using Unsafe

 10. CAS / optimistic locking / lock free

• http://java.dzone.com/articles/lock-based-vs-lock-free - no locks is a lot faster than with locks (no shit).
• http://www.infoq.com/articles/memory_barriers_jvm_concurrency - use volatile but only at your own risk
• 
  

DemiGods in this space - 

• http://mechanical-sympathy.blogspot.com/ - LMAX Tech Architect Guy
• http://vanillajava.blogspot.com/ - HFT Java Guy

JVM Monitoring List

Yes - it's another list. In the last few days, I've had a perverse desire to make lists. This one is for tools that allow for JVM monitoring both internal or external.


Tools:

• http://visualvm.java.net/ - Visual VM
• http://docs.oracle.com/javase/7/docs/technotes/guides/management/jconsole.html - JConsole
• http://www.oracle.com/technetwork/java/javaseproducts/mission-control  - Mission Control 
• http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html
• http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html

Open Source:

• http://code.google.com/p/javasimon/

Vendors:

• http://www.appdynamics.com/
• http://chrononsystems.com/
• http://newrelic.com/java 
• http://www.yourkit.com/ 
• https://profiler.netbeans.org/ 
• http://www.ej-technologies.com/products/jprofiler/overview.html
• http://www.correlsense.com
• http://www.quest.com/foglight-for-java/
• http://www.manageengine.com/products/applications_manager/monitor-jvm-performance.html

• http://www.compuware.com/en_us/application-performance-management/products/purepath-technology.html (DynaTrace)

Metrics:

• https://github.com/Netflix/servo/
• https://github.com/etsy/statsd
• https://github.com/etsy/skyline

Log Aggregation

• http://www.elasticsearch.org/overview/kibana/
• http://www.elasticsearch.org/
• http://logstash.net/
• http://www.splunk.com
• http://sematext.com/logsene/index.html

Tutorials:

• http://www.joyent.com/blog/the-new-systems-performance
• http://dtrace.org/blogs/brendan/2013/06/08/linux-performance-analysis-and-tools/
• http://dtrace.org/blogs/brendan/2012/03/07/the-use-method-linux-performance-checklist/
• http://codeascraft.com/2011/02/15/measure-anything-measure-everything/

Distributed File System List

I feel like making lists. I guess it's a human condition to make lists and organize things. So to satisfy my desire below is a list of distributed file system providers:

Providers:

• http://hadoop.apache.org/docs/stable/hdfs_design.html  - HDFS
• http://fuse.sourceforge.net/  - FUSE
• http://www.xtreemfs.org/  
• http://ceph.com/
• http://www.coda.cs.cmu.edu/
• http://code.google.com/p/kosmosfs/
• https://github.com/quantcast/qfs

 Tutorials:

• http://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/
• http://www.ibm.com/developerworks/library/l-fuse/
• http://www.ibm.com/developerworks/library/wa-introhdfs/

LP Solvers List

I am feeling like making lists. I guess it's the human condition to want to organize and categorize. For this post, I will focus on linear programming solver libraries and links:


Open Source:

• http://www.gnu.org/software/glpk/
• http://sourceforge.net/projects/lpsolve/
• http://commons.apache.org/proper/commons-math/userguide/linear.html
• http://ojalgo.org/
• http://www.coin-or.org/projects/Clp.xml
• http://www.joptimizer.com/
• http://jscience.org/
• http://mikiobraun.github.io/jblas/

 Commercial:

• http://www-01.ibm.com/software/commerce/optimization/cplex-optimizer/
• http://msdn.microsoft.com/en-us/devlabs/hh145003.aspx - Microsoft Solver Foundation
• http://reference.wolfram.com/mathematica/tutorial/ConstrainedOptimizationLinearProgramming.html - Wolfram Mathematica
• http://www.gurobi.com/
• http://www.lindo.com/
• http://www.fico.com/en/Products/DMTools/Pages/FICO-Xpress-Optimization-Suite.aspx
• http://www.mosek.com/ 
• http://www.solver.com/ - Excel Solver
• http://www.ziena.com/
• http://www.ampl.com/
• http://numericalmethod.com/suanshu/
• http://www.ateji.com/optimj/index.html

GC Parameters List

I am always on the lookout for common GC parameters. So I figured I'll compile a list of some of the common parameters and places where I found them:

Super Lists of All Parameters:

1. http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html 
2. http://stas-blogspot.blogspot.fi/2011/07/most-complete-list-of-xx-options-for.html
3. http://blog.ragozin.info/2011/09/hotspot-jvm-garbage-collection-options.html
4. http://reins.altervista.org/java/A_Collection_of_JVM_Options_MP.html 

Blog Roll

• http://practicingtechie.wordpress.com/2013/06/15/java-vm-options/
  
  • -XX:+UseConcMarkSweepGC
  • -XX:+HeapDumpOnOutOfMemoryError
  • -XX:HeapDumpPath=$APP_HOME_DIR
  • -XX:OnOutOfMemoryError=
  • -XX:OnError=
  • -XX:+PrintGCDetails
  • -XX:+PrintGCTimeStamps
  • -Xloggc:$APP_HOME_DIR/gc.log
  • -XX:-UseGCLogFileRotation
  • -XX:GCLogFileSize=M 
  • -XX:NumberOfGCLogFiles=

• http://community.gemstone.com/display/gemfire60/GC+Tuning
  
  • -XX:NewSize=m 
  • -XX:CMSInitiatingOccupancyFraction=N
  • -XX:+UseConcMarkSweepGC
  • -Xmxm
  • -Xmsm

• http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html  
  
  • -XX:+UseConcMarkSweepGC 
  • -XX:ParallelCMSThreads=1  
  • -XX:+CMSIncrementalMode  
  • -XX:+CMSIncrementalPacing  
  • -XX:CMSIncrementalDutyCycleMin=0  
  • -XX:CMSIncrementalDutyCycle=10 

• https://blogs.atlassian.com/2013/02/a-little-practical-gc-tuning-on-eclipse/ 
• -server
• -XX:MaxPermSize=320m
• -XX:+UseParallelGC
• -Xmx3g
• -Xmn2g
• -XX:+DisableExplicitGC

• http://themindstorms.wordpress.com/2009/01/21/advanced-jvm-tuning-for-low-pause/ 
• -Xms2g
• -Xmx2g
• -Xmn150m 
• -XX:GCTimeRatio=2 
• -XX:ParallelGCThreads=8 
• -XX:+UseParNewGC 
• -XX:MaxGCPauseMillis=2000 
• -XX:+DisableExplicitGC

• http://forum.openspaces.org/thread.jspa?messageID=9277 
• -Xms2g 
• -Xmx2g 
• -XX:+UseConcMarkSweepGC 
• -XX:+CMSIncrementalMode 
• -XX:+CMSIncrementalPacing 
• -XX:CMSIncrementalDutyCycleMin=10 
• -XX:CMSIncrementalDutyCycle=50 
• -XX:ParallelGCThreads=8 
• -XX:+UseParNewGC 
• -Xmn150m 
• -XX:MaxGCPauseMillis=2000 
• -XX:GCTimeRatio=10 
• -XX:+DisableExplicitGC 

• http://java-is-the-new-c.blogspot.com/
• -Xms11g 
• -Xmx11g 
• -verbose:gc 
• -XX:-UseAdaptiveSizePolicy 
• -XX:SurvivorRatio=12 
• -XX:NewSize=100m 
• -XX:MaxNewSize=100m 
• -XX:MaxTenuringThreshold=2

• http://techblog.netflix.com/2013/05/garbage-collection-visualization.html
• -verbose:gc
• -verbose:sizes
• -Xloggc:/apps/tomcat/logs/gc.log
• -XX:+PrintGCDetails
• -XX:+PrintGCDateStamps
• -XX:+PrintTenuringDistribution
• -XX:-UseGCOverheadLimit
• -XX:+ExplicitGCInvokesConcurrent
• -XX:+UseConcMarkSweepGC
• -XX:+CMSScavengeBeforeRemark
• -XX:+CMSParallelRemarkEnabled

• https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/5/html/Performance_Tuning_Guide/sect-Performance_Tuning_Guide-Java_Virtual_Machine_Tuning-Garbage_Collection_and_Performance_Tuning.html
• -Xms12288m
• -Xmx12288m
• -XX:+UseLargePages
• -XX:+UseParallelOldGC

• http://onthejvm.blogspot.com/2013/05/java-gc-tuning-for-high-frequency.html
• -XX:+UnlockDiagnosticVMOptions 
• -Xmx4g 
• -Xms4g 
• -XX:MaxNewSize=3g 
• -XX:NewSize=3g 
• -XX:ParallelGCThreads=7 
• -XX:+UseGCTaskAffinity 
• -XX:+BindGCTaskThreadsToCPUs 
• -XX:ParGCCardsPerStrideChunk=32768
  

• http://glipho.com/albert/performance-effects-of-garbage-collection-tuning-on-java
• -XX:+UseConcMarkSweepGC
• -XX:ParallelCMSThreads=8
• -XX:+CMSIncrementalMode
• -XX:+CMSIncrementalPacing
• -XX:CMSIncrementalDutyCycleMin=0
• -XX:CMSIncrementalDutyCycle=10

• http://docs.neo4j.org/chunked/milestone/configuration-jvm.html
• -Xmx
• -XX:+UseConcMarkSweepGC
• -XX:NewRatio=#

• http://blog.igorminar.com/2010/07/dgc-ii-jvm-tuning.html 
• -XX:+UseConcMarkSweepGC
• -XX:+UseParNewGC
• -XX:CMSInitiatingOccupancyFraction=68
• -XX:MaxTenuringThreshold=31
• -XX:+CMSParallelRemarkEnabled
• -XX:SurvivorRatio=6
• -XX:TargetSurvivorRatio=90 
• -XX:+AggressiveOpts
• -XX:+DoEscapeAnalysis 
• -Xloggc:/some/path/
• -XX:+PrintGCDetails 
• -XX:+PrintGCTimeStamps
• -XX:+PrintGCDateStamps
• -XX:+PrintTenuringDistribution
• -XX:+HeapDumpOnOutOfMemoryError
• -Xmn2818m 

• http://blog.performize-it.com/2013/09/jvm-params-everyone-should-have-in.html
• -Xms{#MB}m -Xmx{#MB}m
• -XX:PermSize={#MB}m -XX:MaxPermSize={#MB}m
• -XX:+HeapDumpOnOutOfMemoryError
• -XX:+PrintFlagsFinal
• -server
• -XX:+PrintGCDetails
• -XX:+PrintGCDateStamps 
• -XX:+PrintTenuringDistribution 
• -XX:+PrintGCApplicationStoppedTime 
• -XX:+PrintGCApplicationConcurrentTime  
• -XX:+UseGCLogFileRotation
• -XX:NumberOfGCLogFiles={#files}
• -XX:GCLogFileSize={#MB}M
• -Xloggc:{some gc log file}.gc 
• -Dcom.sun.management.jmxremote
• -Dcom.sun.management.jmxremote.port={a port}
• -Dcom.sun.management.jmxremote.authenticate=false
• -Dcom.sun.management.jmxremote.authenticate=false

• Big Bank System with a very large Heap (~80gb)
• -d64
• -server
• -XX:+AggressiveOpts
• -XX:+UseConcMarkSweepGC
• -XX:+UseParNewGC
• -XX:ParallelGCThreads=4
• -XX:NewRatio=4

 Tools

1. https://github.com/foursquare/heapaudit
2. https://github.com/twitter/jvmgcprof
3. https://github.com/Netflix/gcviz
4. https://github.com/chewiebug/GCViewer

Tutorials

1. http://www.slideshare.net/aszegedi/everything-i-ever-learned-about-jvm-performance-tuning-twitter
2. http://java.dzone.com/articles/how-tame-java-gc-pauses
3. http://blog.ragozin.info/p/garbage-collection.html
4. http://blog.ragozin.info/2011/10/techtalk-garbage-collection-in-java.html
5. https://blogs.oracle.com/jonthecollector/entry/the_second_most_important_gc
6. http://stackoverflow.com/questions/17009961/understanding-the-java-memory-model-and-garbage-collection
7. http://mechanical-sympathy.blogspot.com/2013/07/java-garbage-collection-distilled.html
8. http://blog.mgm-tp.com/2013/03/garbage-collection-tuning/
9. http://www.youtube.com/watch?v=o6qx_zvpOyI  
10. http://www.infoq.com/presentations/Virtualizing-Tuning-JVM
11. https://blog.codecentric.de/en/2013/10/useful-jvm-flags-part-7-cms-collector/
12. http://blog.headius.com/2009/01/my-favorite-hotspot-jvm-flags.html 
13. http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
14. http://java.dzone.com/articles/java-garbage-collection-0 

Greenfield Enterprise Architecture for an IT BU

Let's say you had the power to have a completely greenfield development for an entire Enterprise Architecture for an IT BU - what would it look like?

Well, which IT BU you might ask? Does it really matter?

You need a big database. What kind of enterprise architecture can you have without a database?

Next, we need a bunch of ETL to load data from source systems, because there is always a source system to source from. We can either code it or buy it - doesn't matter which.

And there shall be database performance problems with the data-load.

Next, we need some engine code. Let's call it "The System". And, It shall be in Java. And it shall have memory problems regardless of whether it's 32bit or 64bit or how much memory you allocate to it.

And next, there shall be a web-ui, for which, countless hours will be spent on things that will never be used. And there shall be an excel download link on every page - because that's the only feature the users seem to care about.

And there shall be data quality problems, and performance problems, and scalability problems, and extensibility, and bulk upload requirements, and usability issues, and technical debt, and The Business will cry out for Salvation.

Surely, there must be another way.

I have spent the last 5 years as an Enterprise Architect at a Tier 1 Investment Bank designing systems that solve Big and Expensive problems. There are a few observations I would like to make for things that worked and things that didn't.

1. User's are smart and love Excel and are better at coding than the H1B coder you got for the 2 for 1 sale from a body shop.
2. Build functions not systems - and expose your functions to your users. Allow the users to create a managed ecosystem around the functionality.
3. Make sure your functions work natively in Excel - think COM C# library.
4. Use elastic infrastructure like HDFS, and compute clouds, and data-grids, etc... - don't build dedicated systems, build services that have clean inputs and outputs and can run on scalable hardware like compute clouds.
5. The database and ETL has been my Achilles heel. The database schema is too rigid for the fast pass of change. Alternatively, the rigidness is required given how central the data is too everything. I have yet to really embrace the nosql movement, given the lack of ACID qualities. There are some promising developments in the form of Impala, which is a closer to a pure MPP database running on commodity elastic hardware. Perhaps an interim medium can be found between a strict data-model of a traditional database and a loose schema of a nosql database.

To be continued....

Cross Language

Java to C to C++/CLI to C# to Java via IKVM to HTTP Rest to Java Servlet


and it all works!!!